Re: [Syslog] #2, max message size
Anton, Please look into getting your email program to do word wrapping before 80 columns. Everyone else on the list seems to be able to do this. Replying to an email where you have to manually reformat every line of quoted text is no fun. Thanks. > There is NO maximum message size defined in syslog-protocol draft! .. Then why has there been any discussion, at all, on a maximum message size ? Seems like a waste of time and effort on everyone's behalf, including Rainer's for listing it in the first place. Cheers, Darren ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
Re: [Syslog] #2, max message size
[ Charset ISO-8859-1 unsupported, converting... ] > > Of course, specifying a minimum required support in each transport > mapping is also appropriate. In fact, syslog-transport-udp sets two > separate minimums: for IPv4 and IPv6. If we did not specify a minimum > in syslog-protocol, one can infer the minimum based on the least > common denominator of transport protocol used. That's fair. But I just > don't see a harm in what was done in the current draft (after very > long discussions I might add). I can only a see a very hypothetical > scenario where some future transport protocol has MTU smaller than the > what IPv4 and IPv6 uses. In this case a minimum requirement may be > problematic. How much lower than 480 octets can it go with a > ~100-octet syslog header? I think you mean network protocol, not transport protocol. IPv6 has a larger minimum packet size than IPv4 and unless there is a new IETF backed network layer protocol that has a smaller protocol header than IPv4, I can't see the minimum MTU getting any smaller. Are you suggeting something like syslog over ATM or syslog over PPP as an example ? Darren ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
Re: [Syslog] #2, max message size - Need to resolve this
> I think there is general agreement to specify minimum msg size, not > maximum msg size in syslog-protocol. FWIW, I think this is a much better idea. Darren ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
RE: [Syslog] #2, max message size - Need to resolve this
I think there is general agreement to specify minimum msg size, not maximum msg size in syslog-protocol. Concerning the transport, the same should hold true. I could see that there may be cases in which a transport might specify a minimum msg size that is larger than the one in syslog protocol (so, if syslog protocol is used over a certain transport, message size may be larger than what would be mandated by syslog protocol itself). I don't see that you should mandate to define a max message size for the same reasons we wouldn't define it in syslog-protocol itself. Why unnecessarily impose constraints when you don't have to? In other words, just define min sizes that implementations are obliged to support, but don't prevent them from supporting more if they want to. Just my $0.02. --- Alex -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Andrew Ross Sent: Wednesday, November 30, 2005 2:41 PM To: Chris Lonvick (clonvick); [EMAIL PROTECTED] Subject: RE: [Syslog] #2, max message size - Need to resolve this My vote is for the way Rainer has worded it now. Specify the minimum msg size in syslog-protocol and define max message size in the transport documents. Cheers Andrew Hi Folks, We need to resolve this one. I've heard from Rainer and a very few others. I'd like to hear from more people on this. Choose one: __ The maximum message length needs to be defined in syslog-protocol. __ The maximum message length should be defined in the transport documents. __ I have a different idea Please VOTE NOW! Thanks, Chris ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
RE: [Syslog] #3 NUL octets, #4 binary data, #8 octet-counting
Either solution would work, I have no preference either way. :) Cheers Andrew -Original Message- From: Anton Okmianski (aokmians) [mailto:[EMAIL PROTECTED] Sent: Thursday, 1 December 2005 7:38 a.m. To: [EMAIL PROTECTED]; [EMAIL PROTECTED] Subject: RE: [Syslog] #3 NUL octets, #4 binary data, #8 octet-counting I think for TCP mapping a transport header with message size would be more appropriate framing than termination character. Thanks, Anton. > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Andrew Ross > Sent: Wednesday, November 30, 2005 7:19 AM > To: [EMAIL PROTECTED] > Subject: RE: [Syslog] #3 NUL octets, #4 binary data, #8 octet-counting > > > Rainer, > > That sounds good to me at this stage, and it keeps the door > open. I would prefer to see all binary data encoded in some > "safe" format like base64. It just makes logging the data to > file much easier. For instance, if the binary data contained > a LF character, when it was logged to disk, it would end up > as two separate messages when opened in notepad etc. > > Also, if we are to transport syslog over TCP at some stage, > we need to keep a delimiter character free from use in the > message. Again, a LF would be a good choice for this delimiter. > > Cheers > > Andrew > > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] > On Behalf Of Rainer Gerhards > Sent: Wednesday, 30 November 2005 9:26 p.m. > To: [EMAIL PROTECTED] > Subject: [Syslog] #3 NUL octets, #4 binary data, #8 octet-counting > > > Hi WG, > > I have received notes via private mail telling me there seem > to be some existing (and eventually soon upcoming) valid use > cases for binary data in syslog. I think there is no point in > arguing whether that's fortunate or not. It simply looks like > that's the way it is. I do not like the idea of breaking > existing use cases for syslog (because that will only lead to > implementors ignoring the spec and the story of syslog > inconsistencies continues...). As such, I think we need to > provide at least some minimal support for it (aka "not outlaw it"). > > At first, this implies that NUL octets may be present in the message. > > I propose that we write text that discourages the use of NUL, > but allows it if needed. That text should also allow, but > discourage, a receiver to modify messages containing NUL. > With that, we allow the use case, but do not make it a "show > stopper" for implementing compliant software. This would also > be pretty much in sync with what we currently find in > practice, so it is already expected behaviour. Finally, such > text would caution implementors that when NUL octets are > present, chancs are high that eventually present digitial > signatures will be broken. In my point of view, that's fair > and efficient. > > Chris proposal for #5 (character encoding) also provides an > elegant solution for binary data. We can use something like: > > [enc="binary"] > > or > > [enc="base-64"] > > I do NOT intend to specify this - I think it should be in the > scope of a separate document specifying the use of binary > data. Then would also be the right time to discuss all issues > that arise out of it. For now, I just would like to keep the > door open. > > Finally, I propose to extend Chris format so that the message > size can be conveyed. This has been brought up several times > and I think a clean solution is now obvious: > > [enc="utf-8" lang="en" size="MSG-size-in-octets"] > > MSG-size-in-octets would be the size of the MSG part (just > that!) in octets. Counting just the MSG part is sufficient, > as the rest of the message consists of fields properly > delimited. The size is probably most useful for binary data. > > Please comment. > > Rainer > > ___ > Syslog mailing list > Syslog@lists.ietf.org > https://www1.ietf.org/mailman/listinfo/syslog > > > ___ > Syslog mailing list > Syslog@lists.ietf.org > https://www1.ietf.org/mailman/listinfo/syslog > ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
RE: [Syslog] #3 NUL octets, #4 binary data, #8 octet-counting
>We are still ok with always having UTF-8 in SD values, right? >We need this for foreign usernames. We have discussed this before. Yes, this would work for me. We need to ensure that the SD-IDs are always going to be encoded in a known format. UTF-8 is a good choice. Cheers Andrew ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
RE: [Syslog] #2, max message size - Need to resolve this
My vote is for the way Rainer has worded it now. Specify the minimum msg size in syslog-protocol and define max message size in the transport documents. Cheers Andrew Hi Folks, We need to resolve this one. I've heard from Rainer and a very few others. I'd like to hear from more people on this. Choose one: __ The maximum message length needs to be defined in syslog-protocol. __ The maximum message length should be defined in the transport documents. __ I have a different idea Please VOTE NOW! Thanks, Chris ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
RE: [Syslog] #2, max message size
Rainer, Are you suggesting that these 'options' be done in real-time, or at my-application design time? At my-application design time, I don't necessarily know what size strings will come out For example: I might want to have a simple string that indicates that a specified user logged in at a specified workstation. For things like "username", I might have quite a long dn (1024), and the workstation might also have a very long FQDN (255). Am I to not have this type of a string because it might very clearly be more than 480 characters? The likelihood of these two values being this big is low, so I will probably go ahead. Even during real-time, why should I preemptively truncate. I figure send it. It just might make it through normal (Section 8.3). Trying to add the logic to more intelligently truncate seems unlikely. The instructions in section 6.1 seem to have more to do with internal design than any interoperability. Section 8.3 even says go ahead and try to send the packet, making it even more clear this has little to do with interoperability. It is this 'instructions on how to design your syslog receiver' that I object to being normative. There is no such instructions in other protocols that I have reviewed, and I will readily admit that although I am intimate with dozens there are millions of protocols that I don't know at all. It is these reasons why I recommend that the content of section 6.1/8.3 be moved to the udp-protocol binding, and be included as guidelines and not normative text. Yes, I understand that if it is in this layer, the application writer may never see it. That is just as likely of a risk as any of the others mentioned in this string of emails. All that said, I will bow to the will of the more committed. I really am a very interested observer, far more interested in this standard being written and implemented than I am worried about the standard being written in any specific way. John > > John, > > > I understand the arguments that have been used, yes. I > > understand that in the end there will be a small practical > > MTU when transmitting over UDP (I would like it to be the 4k > > that your experiments determined, not 2K, or worse 480). I > > don't argue with that truth. I very simply fail to see why > > this is a syslog-protocol layer and not a transport-binding > > specification. > > > > Side comment, not necessary for the document at hand: I also > > don't understand what I would do different if I could > > determine (in your words > > negotiate) what the MTU actually was. > > At least three options > - abbreviate things (in human-readable text) > - truncate things you know to be less important > - do application level segmentation > > > > I have a message that > > needs to be sent, it will either make it or not. Knowing a > > dynamically determined MTU is of no value as I can't make my > > message smaller, it is what it is. If it gets fragmented, I > > hope it will be reassembled. If it is big, I hope that the > > receiver offers up a big enough buffer to it's socket > > library. It is all hope at this point. I am ok with hope, I > > just don't want you to limit my ability to hope. > > -protocol-15 does not limit your abilit to hope. It limits your worst > case, because you know what minimum length support you can expect. ;) > > Rainer > > John > > > > > -Original Message- > > > From: Rainer Gerhards [mailto:[EMAIL PROTECTED] > > > Sent: Wednesday, November 30, 2005 11:37 AM > > > To: Moehrke, John (GE Healthcare) > > > Cc: [EMAIL PROTECTED] > > > Subject: RE: [Syslog] #2, max message size > > > > > > John, > > > > > > the issue is the simplex nature of syslog. With syslog (other > > > than with > > > almost all other protocols), you send a message and need to > > > *hope* that > > > the recipient can receive it. There is also no negotiation > > > phase. So you > > > need to send it blindly. For example, if you send an IHE 2K > > > message, but > > > the receiver does just support 1K, you will loose 1K without > > > knowing it. > > > As such, it is vital to know at least a "safe size" that you > > > can use and > > > know that it can be processed (if it makes it to the > recipient). We > > > selected 480 octets because that fits into an unfragmented > > > UDP packet in > > > any case. I know that's too few for IHE, but for network > > > management - an > > > important use case for syslog - this is very often > > sufficient. This is > > > the reason why a simplex protocol like syslog (send it and > > > forget it ;)) > > > should provide some guidance about lower limits. Keep in mind > > > that there > > > is NO upper limit - this is done in the transport mappings and the > > > implementations. So if you need 32K for IHE, that is > perfectly well > > > (-transport-UDP, I think, suppport 64K - but you know the > practical > > > limits). > > > > > > Does this clarify? > > > > > > Rainer > > > > > > > -Original Messag
RE: [Syslog] #2, max message size
John, > I understand the arguments that have been used, yes. I > understand that in the end there will be a small practical > MTU when transmitting over UDP (I would like it to be the 4k > that your experiments determined, not 2K, or worse 480). I > don't argue with that truth. I very simply fail to see why > this is a syslog-protocol layer and not a transport-binding > specification. > > Side comment, not necessary for the document at hand: I also > don't understand what I would do different if I could > determine (in your words > negotiate) what the MTU actually was. At least three options - abbreviate things (in human-readable text) - truncate things you know to be less important - do application level segmentation > I have a message that > needs to be sent, it will either make it or not. Knowing a > dynamically determined MTU is of no value as I can't make my > message smaller, it is what it is. If it gets fragmented, I > hope it will be reassembled. If it is big, I hope that the > receiver offers up a big enough buffer to it's socket > library. It is all hope at this point. I am ok with hope, I > just don't want you to limit my ability to hope. -protocol-15 does not limit your abilit to hope. It limits your worst case, because you know what minimum length support you can expect. ;) Rainer > John > > > -Original Message- > > From: Rainer Gerhards [mailto:[EMAIL PROTECTED] > > Sent: Wednesday, November 30, 2005 11:37 AM > > To: Moehrke, John (GE Healthcare) > > Cc: [EMAIL PROTECTED] > > Subject: RE: [Syslog] #2, max message size > > > > John, > > > > the issue is the simplex nature of syslog. With syslog (other > > than with > > almost all other protocols), you send a message and need to > > *hope* that > > the recipient can receive it. There is also no negotiation > > phase. So you > > need to send it blindly. For example, if you send an IHE 2K > > message, but > > the receiver does just support 1K, you will loose 1K without > > knowing it. > > As such, it is vital to know at least a "safe size" that you > > can use and > > know that it can be processed (if it makes it to the recipient). We > > selected 480 octets because that fits into an unfragmented > > UDP packet in > > any case. I know that's too few for IHE, but for network > > management - an > > important use case for syslog - this is very often > sufficient. This is > > the reason why a simplex protocol like syslog (send it and > > forget it ;)) > > should provide some guidance about lower limits. Keep in mind > > that there > > is NO upper limit - this is done in the transport mappings and the > > implementations. So if you need 32K for IHE, that is perfectly well > > (-transport-UDP, I think, suppport 64K - but you know the practical > > limits). > > > > Does this clarify? > > > > Rainer > > > > > -Original Message- > > > From: Moehrke, John (GE Healthcare) > > [mailto:[EMAIL PROTECTED] > > > Sent: Wednesday, November 30, 2005 6:13 PM > > > To: Rainer Gerhards; Darren Reed > > > Cc: [EMAIL PROTECTED] > > > Subject: RE: [Syslog] #2, max message size > > > > > > Shouldn't the MTU be defined by the binding to the transport? > > > I fail to > > > see why the protocol, unbound to a transport, needs to have > > a limit. > > > > > > > -Original Message- > > > > From: [EMAIL PROTECTED] > > > > [mailto:[EMAIL PROTECTED] On Behalf Of > > Rainer Gerhards > > > > Sent: Wednesday, November 30, 2005 11:01 AM > > > > To: Darren Reed > > > > Cc: [EMAIL PROTECTED] > > > > Subject: RE: [Syslog] #2, max message size > > > > > > > > Darren, > > > > > > > > I violently object your argument and leave it up to the > > > rest of the WG > > > > what should be done. I respect your argument, but I do > not like to > > > > re-iterate this forever. > > > > > > > > One time we have discussed this was in October 2004: > > > > > > > > http://www.syslog.cc/ietf/autoarc/msg01289.html > > > > > > > > If you browse the archive, you will find several other > > > ocasions where > > > > this was discussed. > > > > > > > > What's your actual recommendation? Do not say anything and > > > > leave it the > > > > implementor to guess what is OK? After all, its the > implementor's > > > > problem if the message can not be transmitted. Or do > you prefer to > > > > define an API where the lower layer tells the upper layer what > > > > capabilities it has? Maybe MTU discovery? Or an app-level > > > ack? I think > > > > all of these options have already been discussed during the > > > > past years. > > > > What is now in is a (fragile) consensus, but if only you do > > > > not like it, > > > > I think we should go ahead and leave an unhappy Darren > > > > behind. If only I > > > > insist on it, we can also go ahead and leave an unhappy > > > Rainer behind. > > > > But we need to go ahead! > > > > > > > > Rainer > > > > > > > > > -Original Message- > > > > > From: Darren Reed [mailto:[EMAIL PROTECTED
RE: [Syslog] #2, max message size - Need to resolve this
For obvious reasons, I agree with Steve and Anton. Rainer > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Steve > Chang (schang99) > Sent: Wednesday, November 30, 2005 9:46 PM > To: Anton Okmianski (aokmians); Chris Lonvick (clonvick); > [EMAIL PROTECTED] > Subject: RE: [Syslog] #2, max message size - Need to resolve this > > > I agree with Anton's wording and view. > > Instead of capping the size maximally that a syslog receiver > is to support, it should be the minimum size that it should support. > > Steve > > > -Original Message- > > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] > > On Behalf Of Anton Okmianski (aokmians) > > Sent: Wednesday, November 30, 2005 12:15 PM > > To: Chris Lonvick (clonvick); [EMAIL PROTECTED] > > Subject: RE: [Syslog] #2, max message size - Need to resolve this > > > > I vote for a different idea... As in latest syslog-protocol, define > only > > the minimum message size the receivers is required to accept. I vote > for > > defining it in both. Syslog-protocol defines the least > common agreed > upon > > denominator. Transport defines the minimum that is appropriate for > the > > transport, which can be higher if needed. Thus, if a receiver > implements > > a syslog protocol and a given transport, it has to meet both > requirements. > > > > Anton. > > > > > -Original Message- > > > From: [EMAIL PROTECTED] > > > [mailto:[EMAIL PROTECTED] On Behalf Of Chris Lonvick > > > (clonvick) > > > Sent: Wednesday, November 30, 2005 2:08 PM > > > To: [EMAIL PROTECTED] > > > Subject: [Syslog] #2, max message size - Need to resolve this > > > > > > Hi Folks, > > > > > > We need to resolve this one. I've heard from Rainer and > a very few > > > others. I'd like to hear from more people on this. Choose one: > > > > > > __ The maximum message length needs to be defined in > syslog-protocol. > > > > > > > > > __ The maximum message length should be defined in the transport > > > documents. > > > > > > > > > __ I have a different idea > > > > > > > > > Please VOTE NOW! > > > > > > Thanks, > > > Chris > > > > > > ___ > > > Syslog mailing list > > > Syslog@lists.ietf.org > https://www1.ietf.org/mailman/listinfo/syslog > > > > > > > > ___ > > Syslog mailing list > > Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog > > ___ > Syslog mailing list > Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog > ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
RE: [Syslog] #2, max message size - Need to resolve this
I agree with Anton's wording and view. Instead of capping the size maximally that a syslog receiver is to support, it should be the minimum size that it should support. Steve > -Original Message- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] > On Behalf Of Anton Okmianski (aokmians) > Sent: Wednesday, November 30, 2005 12:15 PM > To: Chris Lonvick (clonvick); [EMAIL PROTECTED] > Subject: RE: [Syslog] #2, max message size - Need to resolve this > > I vote for a different idea... As in latest syslog-protocol, define only > the minimum message size the receivers is required to accept. I vote for > defining it in both. Syslog-protocol defines the least common agreed upon > denominator. Transport defines the minimum that is appropriate for the > transport, which can be higher if needed. Thus, if a receiver implements > a syslog protocol and a given transport, it has to meet both requirements. > > Anton. > > > -Original Message- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] On Behalf Of Chris > > Lonvick (clonvick) > > Sent: Wednesday, November 30, 2005 2:08 PM > > To: [EMAIL PROTECTED] > > Subject: [Syslog] #2, max message size - Need to resolve this > > > > Hi Folks, > > > > We need to resolve this one. I've heard from Rainer and a > > very few others. I'd like to hear from more people on this. > > Choose one: > > > > __ The maximum message length needs to be defined in syslog-protocol. > > > > > > __ The maximum message length should be defined in the transport > > documents. > > > > > > __ I have a different idea > > > > > > Please VOTE NOW! > > > > Thanks, > > Chris > > > > ___ > > Syslog mailing list > > Syslog@lists.ietf.org > > https://www1.ietf.org/mailman/listinfo/syslog > > > > ___ > Syslog mailing list > Syslog@lists.ietf.org > https://www1.ietf.org/mailman/listinfo/syslog ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
RE: [Syslog] #3 NUL octets, #4 binary data, #8 octet-counting
Anton: > Specifying the encoding makes sense to me. This way we can > state that only certain encoding support is required, but not > preclude other options. > > We are still ok with always having UTF-8 in SD values, right? Yes, I was talking exclusively about MSG. I expect SD values to be written by newer software only. There is no POSIX API for it, so we can't have an issue with that API ;) Rainer > We need this for foreign usernames. We have discussed this before. > > Thanks, > Anton. > > > -Original Message- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] On Behalf Of Rainer Gerhards > > Sent: Wednesday, November 30, 2005 3:26 AM > > To: [EMAIL PROTECTED] > > Subject: [Syslog] #3 NUL octets, #4 binary data, #8 octet-counting > > > > Hi WG, > > > > I have received notes via private mail telling me there seem > > to be some existing (and eventually soon upcoming) valid use > > cases for binary data in syslog. I think there is no point in > > arguing whether that's fortunate or not. It simply looks like > > that's the way it is. I do not like the idea of breaking > > existing use cases for syslog (because that will only lead to > > implementors ignoring the spec and the story of syslog > > inconsistencies continues...). As such, I think we need to > > provide at least some minimal support for it (aka "not outlaw it"). > > > > At first, this implies that NUL octets may be present in > the message. > > > > I propose that we write text that discourages the use of NUL, > > but allows it if needed. That text should also allow, but > > discourage, a receiver to modify messages containing NUL. > > With that, we allow the use case, but do not make it a "show > > stopper" for implementing compliant software. This would also > > be pretty much in sync with what we currently find in > > practice, so it is already expected behaviour. Finally, such > > text would caution implementors that when NUL octets are > > present, chancs are high that eventually present digitial > > signatures will be broken. In my point of view, that's fair > > and efficient. > > > > Chris proposal for #5 (character encoding) also provides an > > elegant solution for binary data. We can use something like: > > > > [enc="binary"] > > > > or > > > > [enc="base-64"] > > > > I do NOT intend to specify this - I think it should be in the > > scope of a separate document specifying the use of binary > > data. Then would also be the right time to discuss all issues > > that arise out of it. For now, I just would like to keep the > > door open. > > > > Finally, I propose to extend Chris format so that the message > > size can be conveyed. This has been brought up several times > > and I think a clean solution is now obvious: > > > > [enc="utf-8" lang="en" size="MSG-size-in-octets"] > > > > MSG-size-in-octets would be the size of the MSG part (just > > that!) in octets. Counting just the MSG part is sufficient, > > as the rest of the message consists of fields properly > > delimited. The size is probably most useful for binary data. > > > > Please comment. > > > > Rainer > > > > ___ > > Syslog mailing list > > Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog > > > ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
RE: [Syslog] #2, max message size - Need to resolve this
I vote for a different idea... As in latest syslog-protocol, define only the minimum message size the receivers is required to accept. I vote for defining it in both. Syslog-protocol defines the least common agreed upon denominator. Transport defines the minimum that is appropriate for the transport, which can be higher if needed. Thus, if a receiver implements a syslog protocol and a given transport, it has to meet both requirements. Anton. > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Chris > Lonvick (clonvick) > Sent: Wednesday, November 30, 2005 2:08 PM > To: [EMAIL PROTECTED] > Subject: [Syslog] #2, max message size - Need to resolve this > > Hi Folks, > > We need to resolve this one. I've heard from Rainer and a > very few others. I'd like to hear from more people on this. > Choose one: > > __ The maximum message length needs to be defined in syslog-protocol. > > > __ The maximum message length should be defined in the transport > documents. > > > __ I have a different idea > > > Please VOTE NOW! > > Thanks, > Chris > > ___ > Syslog mailing list > Syslog@lists.ietf.org > https://www1.ietf.org/mailman/listinfo/syslog > ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
RE: [Syslog] #5 - character encoding (was: Consensus?)
Chris:
I agree with all your points. Recommend an encoding and standard lang
tag, and accept all other encoding and lang specification.
Regards,
Sheran
-Original Message-
From: Chris Lonvick (clonvick)
Sent: Wednesday, November 30, 2005 5:06 AM
To: Shyyunn Lin (sheranl)
Cc: [EMAIL PROTECTED]
Subject: RE: [Syslog] #5 - character encoding (was: Consensus?)
Hi Sheran,
On Tue, 29 Nov 2005, Shyyunn Lin (sheranl) wrote:
> Chris:
>
> I think having SD-ID with [enc="utf-8" lang="English"] may be a good
> approach. If different language use utf-8 encoding, then "lang=" can
> distinguish it.
We _should_ be using language codes from RFC 3066. That specifies ISO
639 language tags. 639-1 has 2 character codes ("en" is English) and
639-2 has 3 characters ("eng" is English). RFC 3066 will likely be
replaced by the works of the Language Tag Registry Update (ltru) Working
Group.
http://www.ietf.org/html.charters/ltru-charter.html
They have IDs in the works. Until those become RFCs we should continue
to reference RFC 3066.
>
> Also want to clarify that you suggest that if the message is in ASCII,
> it will not required SD-ID, but for all other encodings, SD-ID will be
> required.
Yes - that's my suggestion.
>
> Note most other encoding methods already imply the language used, for
> example, in Chinese, there are several encoding methods, Traditional
> Chinese used in Taiwan and Hong Kong is Big5, and simplified Chinese
> used in Mainland China is GBK, so if the message is in traditional
> Chinese char, it will be shown as [enc="Big5", lang="Traditional
> Chinese"], a little bit redundant. The Big5 also includes all English
> char so it can be a mix of Chinese and English.
Good point. As far as I can tell, "Big5" is not recognized by any
accredited standards developing organization. It is recognized by the
Ideographic Rapporteur Group (IRG) which reports to the Unicode
consortium. The recognized way to represent Chinese characters,
traditional and simplified, is through ISO 639-2 with the subcodes to
indicate traditional and simplified for the "zh" _language_. The ID on
"Tags for Identifying Languages"
http://www.ietf.org/internet-drafts/draft-ietf-ltru-registry-14.txt
identifies simplified Chinese as "zh-Hans" and traditional Chinese as
"zh-Hant". Additional subtags could identify a locale such as
"zh-Hant-TW" for Taiwan Chinese in traditional script. This is from the
"Initial Language Subtag Registry" ID.
http://www.ietf.org/internet-drafts/draft-ietf-ltru-initial-06.txt
I think that we should specify encoding and language tags as
striaghtforward as possible and let others augment syslog-protocol (in
the
future) with other encoding mechanisms. We can RECOMMEND that encoding
be in UTF-8 and language tags come from RFC 3066. We can allow that
other encoding and language identifications are acceptable. In the
worst case, a vendor will have the option of [EMAIL PROTECTED]"something"
[EMAIL PROTECTED]"piglatin"].
Does this work for you?
Thanks,
Chris
>
>
>
> Regards,
>
> Sheran
>
> -Original Message-
> From: [EMAIL PROTECTED]
> [mailto:[EMAIL PROTECTED] On Behalf Of Chris Lonvick
> (clonvick)
> Sent: Tuesday, November 29, 2005 10:22 AM
> To: Rainer Gerhards
> Cc: [EMAIL PROTECTED]
> Subject: RE: [Syslog] #5 - character encoding (was: Consensus?)
>
> Hi Rainer,
>
> Why don't we look at it from the other direction? We could state that
> any encoding is acceptable - for ease-of-use/migration with existing
> syslog implementations. It is RECOMMENDED that UTF-8 be used. When
> it is used, an SD-ID element will be REQUIRED. e.g. - [enc="utf-8"
> lang="en"]
>
> Thoughts?
>
> All: Let's discuss this and close this issue.
>
> Thanks,
> Chris
>
> On Tue, 29 Nov 2005, Rainer Gerhards wrote:
>
>> Chris & WG,
>>
#5 Character encoding in MSG: due to my proof-of-concept
implementation, I have raised the (ugly) question if we need
to allow encodings other than UTF-8. Please note that this
question arises from needs introduced by e.g. POSIX. So we
can't easily argue them away by whishful thinking ;)
Not even discussed yet.
>>>
>>> I haven't reviewed that yet. However, I'll note that allowing
>>> different encoding can be accomplished in the future as long as we
>>> establish a default encoding and a way to identify it in our current
>>> work.
>>
>> I have read a little in the mailing archive. Please note that in 2000
>> it was consensus that the MSG part may contain encodings other then
>> US-ASCII. Follow this threat:
>>
>> http://www.syslog.cc/ietf/autoarc/msg00127.html
>>
>> This discussion lead to RFC 3164 saying "other encodings MAY be
used".
>> While this was observed behaviour, we need still to be aware that the
>> POSIX (and glibc) API places the restrictions on us that we simply do
>> not know the character encoding used by the application. As such, no
>> *nix syslogd can be programmed to be compliant to syslog-
RE: [Syslog] #2, max message size
Darren: > If you really want to get back to basics, I'd not accept any > maximum message size that was bigger than 490 bytes > (576-14-64-8) as this is the largest frame size that IPv4 is > *required* to reassemble. Either you remove the maximum > message size from syslog-protocol or drop it to 490 but leave > it open to refining by transport definitions. Yes, I'm well > aware of 490 being "too small" for some practical purposes > but you're not thinking clearly if you want any sort of > maximum size in syslog-protocol and this needs to be reinforced. Before saying that somebody is not thinking clearly you should read what they sent you (if not the draft that is being discussed). There is NO maximum message size defined in syslog-protocol draft! Rainer sent you a copy of relevant text as part of this thread too. It only defined a minimum size receivers must be able to process. That's all. A lot of your remarks come across very inflammatory, especially when they reflect that you are have not read the drafts. Could you calm down the rhetoric maybe and do some home work? Every email from you seems to scream that we are bunch of morons who don't know what they are doing, and you have come to save us from our misery. I don't think it is the most productive way to get your arguments across. There are a lot of knowledgeable people in this WG. And thanks for the re-education about the minimum MTU. If you paid any attention to what this WG has done over the last two years or cared to read the archives (which even Rainer has done when discussing issues with you), you would have known that all these issues have been discussed at length here. How about read the syslog-protocol-udp draft (sec 3.2)? Anton. ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
RE: [Syslog] #2, max message size
The draft text does not set a limit on maximum size. It sets a minimum support boundary, which simplifies the basic interop. Is it a problem? The reason for the minimum size to be specified in message protocol and not the transport was to sets the basic expectation for minimum support from transport mappings. For example, if I have a system that fires messages to a sender sub-system and the sender-subsystem can be configured to use a number of transport binding... What is the minimum message I am guaranteed that all receivers will support regardless of the transport used? This information can be critical when you design some important messages. Of course, specifying a minimum required support in each transport mapping is also appropriate. In fact, syslog-transport-udp sets two separate minimums: for IPv4 and IPv6. If we did not specify a minimum in syslog-protocol, one can infer the minimum based on the least common denominator of transport protocol used. That's fair. But I just don't see a harm in what was done in the current draft (after very long discussions I might add). I can only a see a very hypothetical scenario where some future transport protocol has MTU smaller than the what IPv4 and IPv6 uses. In this case a minimum requirement may be problematic. How much lower than 480 octets can it go with a ~100-octet syslog header? Thanks, Anton. > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Rainer Gerhards > Sent: Wednesday, November 30, 2005 12:01 PM > To: Darren Reed > Cc: [EMAIL PROTECTED] > Subject: RE: [Syslog] #2, max message size > > Darren, > > I violently object your argument and leave it up to the rest > of the WG what should be done. I respect your argument, but I > do not like to re-iterate this forever. > > One time we have discussed this was in October 2004: > > http://www.syslog.cc/ietf/autoarc/msg01289.html > > If you browse the archive, you will find several other > ocasions where this was discussed. > > What's your actual recommendation? Do not say anything and > leave it the implementor to guess what is OK? After all, its > the implementor's problem if the message can not be > transmitted. Or do you prefer to define an API where the > lower layer tells the upper layer what capabilities it has? > Maybe MTU discovery? Or an app-level ack? I think all of > these options have already been discussed during the past years. > What is now in is a (fragile) consensus, but if only you do > not like it, I think we should go ahead and leave an unhappy > Darren behind. If only I insist on it, we can also go ahead > and leave an unhappy Rainer behind. > But we need to go ahead! > > Rainer > > > -Original Message- > > From: Darren Reed [mailto:[EMAIL PROTECTED] > > Sent: Wednesday, November 30, 2005 4:49 PM > > To: Rainer Gerhards > > Cc: [EMAIL PROTECTED] > > Subject: Re: [Syslog] #2, max message size > > > > > Darren, > > > > > > > The only place a message size limit should be specified is in a > > > > transport mapping. If it's in -15 then it should be removed. > > > > Limits of all sizes and types do nothing but contribute > to aging > > > > of a protocol. > > > > > > -protocol-15 is a compromise after a very long > discussion. It says: > > > > > > - > > >A receiver MUST be able to accept messages up to and > > including 480 > > >octets in length. For interoperability reasons, all receiver > > >implementations SHOULD be able to accept messages up to > > and including > > >2,048 octets in length. > > > > > >If a receiver receives a message with a length larger > than 2,048 > > >octets, or larger than it supports, the receiver MAY > discard the > > >message or truncate the payload. > > > - > > > > > > I think this text is useful. It keeps the door open for any size > > > messages while still allowing it to be restricted by the > transport > > > mappings and individual implementations (e.g. on low-end embedded > > > devices). It cautions implementors against being too > > verbose but also > > > sets a lower limit that each implementation can assume to > > be received. > > > > > > I think we should continue to use this text. Do you agree? > > > > No. That text doesn't belong in this draft. > > > > Darren > > > > ___ > Syslog mailing list > Syslog@lists.ietf.org > https://www1.ietf.org/mailman/listinfo/syslog > ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
Re: [Syslog] #7 field order
> WG, > > there has not been much discussion about the header fields and their > order recently. I think this is a sign the issue has been settled. To > make sure I got the right understanding of the resulting consensus, I > propose that we use the following format: > > VERSION SP TIMESTAMP SP HOSTNAME SP APP-NAME SP PROCID SP MSGID SP > [SD-ID]s SP MSG > > That is the format that also proven to be quite useful during my > proof-of-concept implementation. > > If somebody objects, please do that now. In your other email, you say that "-" is for a missing field - I was curious about whether all fields were mandatory or what was to be done if you wanted to miss one out, but you've answered that. The "HOSTNAME" field should be constrained, in its definition, to match that accepted for FQDNs. "PRINTUSASCII" is too wide. I believe you need to read RFC 1035. Similarly, I'd like to see APP-NAME, PROCID and MSGID refined to be less than the entire character set. A contradiction in syslog-protocol is allowing PRINTUSASCII for fields but a field of "-" is used to indicate it is not there. The only thing I would like to see considered is keeping a ':' to mark end of headers and start of message. Or is this pointless ? I'm thinking from a perspective of parsing them with awk/grep. If, for example, I can search for either ']: ' or '-: ' and know that what follows would be the messageis that sensible ? Darren ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
Re: [Syslog] #2, max message size - Need to resolve this
--On Wednesday, November 30, 2005 11:07:40 AM -0800 Chris Lonvick <[EMAIL PROTECTED]> wrote: Hi Folks, We need to resolve this one. I've heard from Rainer and a very few others. I'd like to hear from more people on this. Choose one: ... __ The maximum message length should be defined in the transport documents. ... It's a transport issue. We may want to specify a "minimum maximum" - i.e. a syslog receiver MUST be able to handle messages of at least n octets, but that probably also belongs in the transport doc. -- Carson ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
[Syslog] #2, max message size - Need to resolve this
Hi Folks, We need to resolve this one. I've heard from Rainer and a very few others. I'd like to hear from more people on this. Choose one: __ The maximum message length needs to be defined in syslog-protocol. __ The maximum message length should be defined in the transport documents. __ I have a different idea Please VOTE NOW! Thanks, Chris ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
Re: [Syslog] #3 NUL octets, #4 binary data, #8 octet-counting
> Darren, > > > > > Also, if we are to transport syslog over TCP at some stage, > > > > we need to keep > > > > a delimiter character free from use in the message. Again, a > > > > LF would be a > > > > good choice for this delimiter. > > > > > > Here, I disagree. I think we can not set aside a character > > for this. If > > > we go for TCP, let's do octet-couting. Its reliable, efficient and > > > proven. Anyhow, we are not yet doing a TCP mapping, so I > > suggest we save > > > this discussion until later. > > > > Why not use LF? > > Because we - the WG - have said we want to have the full character > set available in the message. Search the archive. Read the drafts. I am > tired of doing (re)search for you. I have better things to do than to > support your lazyness. I don't see why having a full character set available in a message means that you can't chose to select a particular character for an EOR marker. Protocols everywhere use special characters for this or that yet there are no restrictions on what messages they convey. Maybe I'm confusing implementation detail with protocol specification by asking for LF be used as a EOR marker. So, you're right, we shouldn't set aside any particular character to mean EOR in syslog-protocol, that should be left up to the construction of syslog over each protocol. Do you agree on that? > My preference is to finish *this* work. It doesn't help to restart this > discussion. May I provide a quote from Marshall T. Rose in a similar > situation in this WG: Marshall T Rose is responsible for 3195 and that could be considered to be a failure by many. > How about you and your experience and affiliation? I was the focal point of a lot of people wanting to do work on the syslog protocol before the WG formed. At the first syslog IETF BOF I gave a small presentation on the work I'd done to implement syslog over TCP, including using SSL as the transport with the transfer of hashes and storing them on disk. At the first BOF there was a large room full of a lot of people, a far cry from the numbers in Vancouver. The syslog replacement I wrote over 5 years ago was the basis of the work that has resulted in syslog-ng for Linux, even though today it no longer even closely resembles what I did originally. I, like many I suspect, became disenchanted with the direction of the group when it started to go down the road that led to 3195 and I just lost interest for a long time because I did not have the energy to put into fighitng it. In this group, I represent myself. Darren ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
RE: [Syslog] #2, max message size
I understand the arguments that have been used, yes. I understand that in the end there will be a small practical MTU when transmitting over UDP (I would like it to be the 4k that your experiments determined, not 2K, or worse 480). I don't argue with that truth. I very simply fail to see why this is a syslog-protocol layer and not a transport-binding specification. Side comment, not necessary for the document at hand: I also don't understand what I would do different if I could determine (in your words negotiate) what the MTU actually was. I have a message that needs to be sent, it will either make it or not. Knowing a dynamically determined MTU is of no value as I can't make my message smaller, it is what it is. If it gets fragmented, I hope it will be reassembled. If it is big, I hope that the receiver offers up a big enough buffer to it's socket library. It is all hope at this point. I am ok with hope, I just don't want you to limit my ability to hope. John > -Original Message- > From: Rainer Gerhards [mailto:[EMAIL PROTECTED] > Sent: Wednesday, November 30, 2005 11:37 AM > To: Moehrke, John (GE Healthcare) > Cc: [EMAIL PROTECTED] > Subject: RE: [Syslog] #2, max message size > > John, > > the issue is the simplex nature of syslog. With syslog (other > than with > almost all other protocols), you send a message and need to > *hope* that > the recipient can receive it. There is also no negotiation > phase. So you > need to send it blindly. For example, if you send an IHE 2K > message, but > the receiver does just support 1K, you will loose 1K without > knowing it. > As such, it is vital to know at least a "safe size" that you > can use and > know that it can be processed (if it makes it to the recipient). We > selected 480 octets because that fits into an unfragmented > UDP packet in > any case. I know that's too few for IHE, but for network > management - an > important use case for syslog - this is very often sufficient. This is > the reason why a simplex protocol like syslog (send it and > forget it ;)) > should provide some guidance about lower limits. Keep in mind > that there > is NO upper limit - this is done in the transport mappings and the > implementations. So if you need 32K for IHE, that is perfectly well > (-transport-UDP, I think, suppport 64K - but you know the practical > limits). > > Does this clarify? > > Rainer > > > -Original Message- > > From: Moehrke, John (GE Healthcare) > [mailto:[EMAIL PROTECTED] > > Sent: Wednesday, November 30, 2005 6:13 PM > > To: Rainer Gerhards; Darren Reed > > Cc: [EMAIL PROTECTED] > > Subject: RE: [Syslog] #2, max message size > > > > Shouldn't the MTU be defined by the binding to the transport? > > I fail to > > see why the protocol, unbound to a transport, needs to have > a limit. > > > > > -Original Message- > > > From: [EMAIL PROTECTED] > > > [mailto:[EMAIL PROTECTED] On Behalf Of > Rainer Gerhards > > > Sent: Wednesday, November 30, 2005 11:01 AM > > > To: Darren Reed > > > Cc: [EMAIL PROTECTED] > > > Subject: RE: [Syslog] #2, max message size > > > > > > Darren, > > > > > > I violently object your argument and leave it up to the > > rest of the WG > > > what should be done. I respect your argument, but I do not like to > > > re-iterate this forever. > > > > > > One time we have discussed this was in October 2004: > > > > > > http://www.syslog.cc/ietf/autoarc/msg01289.html > > > > > > If you browse the archive, you will find several other > > ocasions where > > > this was discussed. > > > > > > What's your actual recommendation? Do not say anything and > > > leave it the > > > implementor to guess what is OK? After all, its the implementor's > > > problem if the message can not be transmitted. Or do you prefer to > > > define an API where the lower layer tells the upper layer what > > > capabilities it has? Maybe MTU discovery? Or an app-level > > ack? I think > > > all of these options have already been discussed during the > > > past years. > > > What is now in is a (fragile) consensus, but if only you do > > > not like it, > > > I think we should go ahead and leave an unhappy Darren > > > behind. If only I > > > insist on it, we can also go ahead and leave an unhappy > > Rainer behind. > > > But we need to go ahead! > > > > > > Rainer > > > > > > > -Original Message- > > > > From: Darren Reed [mailto:[EMAIL PROTECTED] > > > > Sent: Wednesday, November 30, 2005 4:49 PM > > > > To: Rainer Gerhards > > > > Cc: [EMAIL PROTECTED] > > > > Subject: Re: [Syslog] #2, max message size > > > > > > > > > Darren, > > > > > > > > > > > The only place a message size limit should be > specified is in > > > > > > a transport > > > > > > mapping. If it's in -15 then it should be removed. Limits > > > > > > of all sizes > > > > > > and types do nothing but contribute to aging of a protocol. > > > > > > > > > > -protocol-15 is a compromise after a very long > > > discussion. It say
RE: [Syslog] #3 NUL octets, #4 binary data, #8 octet-counting
I think for TCP mapping a transport header with message size would be more appropriate framing than termination character. Thanks, Anton. > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Andrew Ross > Sent: Wednesday, November 30, 2005 7:19 AM > To: [EMAIL PROTECTED] > Subject: RE: [Syslog] #3 NUL octets, #4 binary data, #8 octet-counting > > > Rainer, > > That sounds good to me at this stage, and it keeps the door > open. I would prefer to see all binary data encoded in some > "safe" format like base64. It just makes logging the data to > file much easier. For instance, if the binary data contained > a LF character, when it was logged to disk, it would end up > as two separate messages when opened in notepad etc. > > Also, if we are to transport syslog over TCP at some stage, > we need to keep a delimiter character free from use in the > message. Again, a LF would be a good choice for this delimiter. > > Cheers > > Andrew > > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] > On Behalf Of Rainer Gerhards > Sent: Wednesday, 30 November 2005 9:26 p.m. > To: [EMAIL PROTECTED] > Subject: [Syslog] #3 NUL octets, #4 binary data, #8 octet-counting > > > Hi WG, > > I have received notes via private mail telling me there seem > to be some existing (and eventually soon upcoming) valid use > cases for binary data in syslog. I think there is no point in > arguing whether that's fortunate or not. It simply looks like > that's the way it is. I do not like the idea of breaking > existing use cases for syslog (because that will only lead to > implementors ignoring the spec and the story of syslog > inconsistencies continues...). As such, I think we need to > provide at least some minimal support for it (aka "not outlaw it"). > > At first, this implies that NUL octets may be present in the message. > > I propose that we write text that discourages the use of NUL, > but allows it if needed. That text should also allow, but > discourage, a receiver to modify messages containing NUL. > With that, we allow the use case, but do not make it a "show > stopper" for implementing compliant software. This would also > be pretty much in sync with what we currently find in > practice, so it is already expected behaviour. Finally, such > text would caution implementors that when NUL octets are > present, chancs are high that eventually present digitial > signatures will be broken. In my point of view, that's fair > and efficient. > > Chris proposal for #5 (character encoding) also provides an > elegant solution for binary data. We can use something like: > > [enc="binary"] > > or > > [enc="base-64"] > > I do NOT intend to specify this - I think it should be in the > scope of a separate document specifying the use of binary > data. Then would also be the right time to discuss all issues > that arise out of it. For now, I just would like to keep the > door open. > > Finally, I propose to extend Chris format so that the message > size can be conveyed. This has been brought up several times > and I think a clean solution is now obvious: > > [enc="utf-8" lang="en" size="MSG-size-in-octets"] > > MSG-size-in-octets would be the size of the MSG part (just > that!) in octets. Counting just the MSG part is sufficient, > as the rest of the message consists of fields properly > delimited. The size is probably most useful for binary data. > > Please comment. > > Rainer > > ___ > Syslog mailing list > Syslog@lists.ietf.org > https://www1.ietf.org/mailman/listinfo/syslog > > > ___ > Syslog mailing list > Syslog@lists.ietf.org > https://www1.ietf.org/mailman/listinfo/syslog > ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
Re: [Syslog] #2, max message size
> John, > > the issue is the simplex nature of syslog. With syslog (other than with > almost all other protocols), you send a message and need to *hope* that > the recipient can receive it. There is also no negotiation phase. So you > need to send it blindly. What you're failing to grasp here is that syslog-protocol, by itself, does not define how it is to be used. All it is doing is defining a message format. With a layered approach to defining syslog we need to split out documentation that is specific to an implementation (such as the maximum message size) from the definition of the protocol messages themselves. In your charter you have a "syslog-protocol over UDP". So any talk in syslog-protocol about maximum message size is going to be redundant for UDP. Using syslog-protocol over TCP is not yet on the radar so anyone making assumptions about maximum size is doing so at their own risk. If they choose 4K or 8K or 1K as the maximum size, that's up to them. If you really want to get back to basics, I'd not accept any maximum message size that was bigger than 490 bytes (576-14-64-8) as this is the largest frame size that IPv4 is *required* to reassemble. Either you remove the maximum message size from syslog-protocol or drop it to 490 but leave it open to refining by transport definitions. Yes, I'm well aware of 490 being "too small" for some practical purposes but you're not thinking clearly if you want any sort of maximum size in syslog-protocol and this needs to be reinforced. Darren ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
RE: [Syslog] #3 NUL octets, #4 binary data, #8 octet-counting
Specifying the encoding makes sense to me. This way we can state that only certain encoding support is required, but not preclude other options. We are still ok with always having UTF-8 in SD values, right? We need this for foreign usernames. We have discussed this before. Thanks, Anton. > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Rainer Gerhards > Sent: Wednesday, November 30, 2005 3:26 AM > To: [EMAIL PROTECTED] > Subject: [Syslog] #3 NUL octets, #4 binary data, #8 octet-counting > > Hi WG, > > I have received notes via private mail telling me there seem > to be some existing (and eventually soon upcoming) valid use > cases for binary data in syslog. I think there is no point in > arguing whether that's fortunate or not. It simply looks like > that's the way it is. I do not like the idea of breaking > existing use cases for syslog (because that will only lead to > implementors ignoring the spec and the story of syslog > inconsistencies continues...). As such, I think we need to > provide at least some minimal support for it (aka "not outlaw it"). > > At first, this implies that NUL octets may be present in the message. > > I propose that we write text that discourages the use of NUL, > but allows it if needed. That text should also allow, but > discourage, a receiver to modify messages containing NUL. > With that, we allow the use case, but do not make it a "show > stopper" for implementing compliant software. This would also > be pretty much in sync with what we currently find in > practice, so it is already expected behaviour. Finally, such > text would caution implementors that when NUL octets are > present, chancs are high that eventually present digitial > signatures will be broken. In my point of view, that's fair > and efficient. > > Chris proposal for #5 (character encoding) also provides an > elegant solution for binary data. We can use something like: > > [enc="binary"] > > or > > [enc="base-64"] > > I do NOT intend to specify this - I think it should be in the > scope of a separate document specifying the use of binary > data. Then would also be the right time to discuss all issues > that arise out of it. For now, I just would like to keep the > door open. > > Finally, I propose to extend Chris format so that the message > size can be conveyed. This has been brought up several times > and I think a clean solution is now obvious: > > [enc="utf-8" lang="en" size="MSG-size-in-octets"] > > MSG-size-in-octets would be the size of the MSG part (just > that!) in octets. Counting just the MSG part is sufficient, > as the rest of the message consists of fields properly > delimited. The size is probably most useful for binary data. > > Please comment. > > Rainer > > ___ > Syslog mailing list > Syslog@lists.ietf.org > https://www1.ietf.org/mailman/listinfo/syslog > ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
RE: [Syslog] #2, max message size
John, the issue is the simplex nature of syslog. With syslog (other than with almost all other protocols), you send a message and need to *hope* that the recipient can receive it. There is also no negotiation phase. So you need to send it blindly. For example, if you send an IHE 2K message, but the receiver does just support 1K, you will loose 1K without knowing it. As such, it is vital to know at least a "safe size" that you can use and know that it can be processed (if it makes it to the recipient). We selected 480 octets because that fits into an unfragmented UDP packet in any case. I know that's too few for IHE, but for network management - an important use case for syslog - this is very often sufficient. This is the reason why a simplex protocol like syslog (send it and forget it ;)) should provide some guidance about lower limits. Keep in mind that there is NO upper limit - this is done in the transport mappings and the implementations. So if you need 32K for IHE, that is perfectly well (-transport-UDP, I think, suppport 64K - but you know the practical limits). Does this clarify? Rainer > -Original Message- > From: Moehrke, John (GE Healthcare) [mailto:[EMAIL PROTECTED] > Sent: Wednesday, November 30, 2005 6:13 PM > To: Rainer Gerhards; Darren Reed > Cc: [EMAIL PROTECTED] > Subject: RE: [Syslog] #2, max message size > > Shouldn't the MTU be defined by the binding to the transport? > I fail to > see why the protocol, unbound to a transport, needs to have a limit. > > > -Original Message- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] On Behalf Of Rainer Gerhards > > Sent: Wednesday, November 30, 2005 11:01 AM > > To: Darren Reed > > Cc: [EMAIL PROTECTED] > > Subject: RE: [Syslog] #2, max message size > > > > Darren, > > > > I violently object your argument and leave it up to the > rest of the WG > > what should be done. I respect your argument, but I do not like to > > re-iterate this forever. > > > > One time we have discussed this was in October 2004: > > > > http://www.syslog.cc/ietf/autoarc/msg01289.html > > > > If you browse the archive, you will find several other > ocasions where > > this was discussed. > > > > What's your actual recommendation? Do not say anything and > > leave it the > > implementor to guess what is OK? After all, its the implementor's > > problem if the message can not be transmitted. Or do you prefer to > > define an API where the lower layer tells the upper layer what > > capabilities it has? Maybe MTU discovery? Or an app-level > ack? I think > > all of these options have already been discussed during the > > past years. > > What is now in is a (fragile) consensus, but if only you do > > not like it, > > I think we should go ahead and leave an unhappy Darren > > behind. If only I > > insist on it, we can also go ahead and leave an unhappy > Rainer behind. > > But we need to go ahead! > > > > Rainer > > > > > -Original Message- > > > From: Darren Reed [mailto:[EMAIL PROTECTED] > > > Sent: Wednesday, November 30, 2005 4:49 PM > > > To: Rainer Gerhards > > > Cc: [EMAIL PROTECTED] > > > Subject: Re: [Syslog] #2, max message size > > > > > > > Darren, > > > > > > > > > The only place a message size limit should be specified is in > > > > > a transport > > > > > mapping. If it's in -15 then it should be removed. Limits > > > > > of all sizes > > > > > and types do nothing but contribute to aging of a protocol. > > > > > > > > -protocol-15 is a compromise after a very long > > discussion. It says: > > > > > > > > - > > > >A receiver MUST be able to accept messages up to and > > > including 480 > > > >octets in length. For interoperability reasons, all receiver > > > >implementations SHOULD be able to accept messages up to > > > and including > > > >2,048 octets in length. > > > > > > > >If a receiver receives a message with a length larger > > than 2,048 > > > >octets, or larger than it supports, the receiver MAY > > discard the > > > >message or truncate the payload. > > > > - > > > > > > > > I think this text is useful. It keeps the door open for any size > > > > messages while still allowing it to be restricted by > the transport > > > > mappings and individual implementations (e.g. on > low-end embedded > > > > devices). It cautions implementors against being too > > > verbose but also > > > > sets a lower limit that each implementation can assume to > > > be received. > > > > > > > > I think we should continue to use this text. Do you agree? > > > > > > No. That text doesn't belong in this draft. > > > > > > Darren > > > > > > > ___ > > Syslog mailing list > > Syslog@lists.ietf.org > > https://www1.ietf.org/mailman/listinfo/syslog > > > ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
RE: [Syslog] #3 NUL octets, #4 binary data, #8 octet-counting
Darren, sorry, but this one calls for a very blunt and direct answer... All others, this is flamish, but I can no longer stand it. Enough is enough. Complaints and flames please to me personally. Rainer > -Original Message- > From: Darren Reed [mailto:[EMAIL PROTECTED] > Sent: Wednesday, November 30, 2005 4:48 PM > To: Rainer Gerhards > Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED] > Subject: Re: [Syslog] #3 NUL octets, #4 binary data, #8 octet-counting > > > Andrew, > > > > > Also, if we are to transport syslog over TCP at some stage, > > > we need to keep > > > a delimiter character free from use in the message. Again, a > > > LF would be a > > > good choice for this delimiter. > > > > Here, I disagree. I think we can not set aside a character > for this. If > > we go for TCP, let's do octet-couting. Its reliable, efficient and > > proven. Anyhow, we are not yet doing a TCP mapping, so I > suggest we save > > this discussion until later. > > Why not use LF? Because we - the WG - have said we want to have the full character set available in the message. Search the archive. Read the drafts. I am tired of doing (re)search for you. I have better things to do than to support your lazyness. The past days, you've brought up a lot of arguments, declined proper queries on your own research (which seems to be non-existing, by the way), thrown in vaporware and shown that you do not know the contents of the drafts. Can you tell me a single reason why I should take your seriously? Ignorance is not a good replacement for evidence. > It will work. There are syslogd implementations about that use LF as > a record delimeter. In other emails you're saying "we must support > binary because some people are going to use this". Now we've > got people > that support LF as a record delimeter already but you don't want to > take this into account. If we change that, we need to change the whole idea. Tell me a single implementation that relies on LF as terminator. On UDP. I know it is done in TCP, but that's a different story, because there it is not part of the message but part of the framing (I guess you understand the difference?). TCP needs to be redefined in any way. So, once again, would you please come up at least once with evidence for your claims? > > Why does what one vendor is going to do mean more than what other > vendors are already doing ? Is there bias here somewhere ? Do you > have any particular preference based on work you've done or through > some sort of affiliation ? My preference is to finish *this* work. It doesn't help to restart this discussion. May I provide a quote from Marshall T. Rose in a similar situation in this WG: ### to carry your argument to its logical conclusion, no working group would ever actually produce any final document, because someone would always be free to introduce additional requirements for a just and noble cause. the ietf is about engineering, not perfection. engineering is about bounding a problem, looking at the tradeoffs, and coming up with a solution. the ietf is about fairness, not perfection. fairness is about providing an process that has transparency and closure. the time to raise these issues was when the group was developing the charter and negotiating it with the iesg... /mtr ### (see http://www.syslog.cc/ietf/autoarc/msg00336.html, Jan, 6 2001) Rergarding my affiliation and past experience: I have actually *implemented* EVERYTHING that currently exists in syslog. Even RFC 3195. Both on Windows and Linux. How about you and your experience and affiliation? [you even quote code without knowing how the iov[] is handled later on in F_FORW... poor guy...] > If you want to discount use of LF as a record delimeter then there > is no reason we must support use of binary because what vendors are > doing is obviously irrelevant. > > Darren > ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
RE: [Syslog] #2, max message size
Darren, I violently object your argument and leave it up to the rest of the WG what should be done. I respect your argument, but I do not like to re-iterate this forever. One time we have discussed this was in October 2004: http://www.syslog.cc/ietf/autoarc/msg01289.html If you browse the archive, you will find several other ocasions where this was discussed. What's your actual recommendation? Do not say anything and leave it the implementor to guess what is OK? After all, its the implementor's problem if the message can not be transmitted. Or do you prefer to define an API where the lower layer tells the upper layer what capabilities it has? Maybe MTU discovery? Or an app-level ack? I think all of these options have already been discussed during the past years. What is now in is a (fragile) consensus, but if only you do not like it, I think we should go ahead and leave an unhappy Darren behind. If only I insist on it, we can also go ahead and leave an unhappy Rainer behind. But we need to go ahead! Rainer > -Original Message- > From: Darren Reed [mailto:[EMAIL PROTECTED] > Sent: Wednesday, November 30, 2005 4:49 PM > To: Rainer Gerhards > Cc: [EMAIL PROTECTED] > Subject: Re: [Syslog] #2, max message size > > > Darren, > > > > > The only place a message size limit should be specified is in > > > a transport > > > mapping. If it's in -15 then it should be removed. Limits > > > of all sizes > > > and types do nothing but contribute to aging of a protocol. > > > > -protocol-15 is a compromise after a very long discussion. It says: > > > > - > >A receiver MUST be able to accept messages up to and > including 480 > >octets in length. For interoperability reasons, all receiver > >implementations SHOULD be able to accept messages up to > and including > >2,048 octets in length. > > > >If a receiver receives a message with a length larger than 2,048 > >octets, or larger than it supports, the receiver MAY discard the > >message or truncate the payload. > > - > > > > I think this text is useful. It keeps the door open for any size > > messages while still allowing it to be restricted by the transport > > mappings and individual implementations (e.g. on low-end embedded > > devices). It cautions implementors against being too > verbose but also > > sets a lower limit that each implementation can assume to > be received. > > > > I think we should continue to use this text. Do you agree? > > No. That text doesn't belong in this draft. > > Darren > ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
RE: [Syslog] #2, max message size
Shouldn't the MTU be defined by the binding to the transport? I fail to see why the protocol, unbound to a transport, needs to have a limit. > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Rainer Gerhards > Sent: Wednesday, November 30, 2005 11:01 AM > To: Darren Reed > Cc: [EMAIL PROTECTED] > Subject: RE: [Syslog] #2, max message size > > Darren, > > I violently object your argument and leave it up to the rest of the WG > what should be done. I respect your argument, but I do not like to > re-iterate this forever. > > One time we have discussed this was in October 2004: > > http://www.syslog.cc/ietf/autoarc/msg01289.html > > If you browse the archive, you will find several other ocasions where > this was discussed. > > What's your actual recommendation? Do not say anything and > leave it the > implementor to guess what is OK? After all, its the implementor's > problem if the message can not be transmitted. Or do you prefer to > define an API where the lower layer tells the upper layer what > capabilities it has? Maybe MTU discovery? Or an app-level ack? I think > all of these options have already been discussed during the > past years. > What is now in is a (fragile) consensus, but if only you do > not like it, > I think we should go ahead and leave an unhappy Darren > behind. If only I > insist on it, we can also go ahead and leave an unhappy Rainer behind. > But we need to go ahead! > > Rainer > > > -Original Message- > > From: Darren Reed [mailto:[EMAIL PROTECTED] > > Sent: Wednesday, November 30, 2005 4:49 PM > > To: Rainer Gerhards > > Cc: [EMAIL PROTECTED] > > Subject: Re: [Syslog] #2, max message size > > > > > Darren, > > > > > > > The only place a message size limit should be specified is in > > > > a transport > > > > mapping. If it's in -15 then it should be removed. Limits > > > > of all sizes > > > > and types do nothing but contribute to aging of a protocol. > > > > > > -protocol-15 is a compromise after a very long > discussion. It says: > > > > > > - > > >A receiver MUST be able to accept messages up to and > > including 480 > > >octets in length. For interoperability reasons, all receiver > > >implementations SHOULD be able to accept messages up to > > and including > > >2,048 octets in length. > > > > > >If a receiver receives a message with a length larger > than 2,048 > > >octets, or larger than it supports, the receiver MAY > discard the > > >message or truncate the payload. > > > - > > > > > > I think this text is useful. It keeps the door open for any size > > > messages while still allowing it to be restricted by the transport > > > mappings and individual implementations (e.g. on low-end embedded > > > devices). It cautions implementors against being too > > verbose but also > > > sets a lower limit that each implementation can assume to > > be received. > > > > > > I think we should continue to use this text. Do you agree? > > > > No. That text doesn't belong in this draft. > > > > Darren > > > > ___ > Syslog mailing list > Syslog@lists.ietf.org > https://www1.ietf.org/mailman/listinfo/syslog > ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
Re: [Syslog] #3 NUL octets, #4 binary data, #8 octet-counting
> Darren, .. > So why this gap between 3164 and practice? I have come > to the conclusion that after Eric invented it, other folks have redone > his work on other platforms (e.g. Linux, Windows and of course embedded > devices). While all of these implementors had Eric's ideas on their > mind, there was no spec to follow. Thus, each one introduced subtle > differences and finally even BSD syslogd was modified in subtle ways. At > the time 3164 was defined, nobody went to the lab and verified the > results. Nor did I when I started with syslog-protocol. We all were so > convinced that when Eric agreed to the document, it must be right. I > think we do not need to put finger at anyone. .. I agree with the summary here. > In general, I agree. But I think we should support the currently > existing use cases for syslog. After all, was this effort not put on > hold because it was said it is not backwards-compatible enough? Indeed. But I think there were (or are) protocol subtlties that many people are unaware of. Maybe in your testing of syslog messages you should include sending a message that is 255 characters long 0x1 - 0xff and see what comes out the other end. I'd also recommend that you include Solaris (preferably Solaris 10) in your test suite if you can't include any of HP-UX or AIX. It's probably the most common commercial Unix and people who deploy it are loathe to replace system components with open source bits. Solaris 10 for x86 is a free download. > > > MSG-size-in-octets would be the size of the MSG part (just that!) in > > > octets. Counting just the MSG part is sufficient, as the rest of the > > > message consists of fields properly delimited. The size is > > probably most > > > useful for binary data. > > > > What happens if the message is truncated? > > Good question. I'd say the size would need to be adjusted. What happens if/when the SD data is truncated and either the MSG size is lost or truncated half way through ? I believe there is very little added value of having the message size as part of SD data - or at least not enough to warrant it being treated as a mandatory/required field to include. > > What value is the message size really providing here? > > If we have a natural EOR marker, LF, what do we need the size for? > > We do NOT have a EOR marker. As of the current draft, LF is just a > regular character like any other. We explicitely wanted the capability > to surface, now we have it. I even think its not bad to have it. I do > NOT like the idea to re-iterate that long discussion of whether or not > full UTF-8 should be supported. Please review the mailing list archive > for that. Allowing LF inside a native message is bad, now that I think of it. Supporting UTF-8 doesn't impact what we use as an EOR marker unless we are going to say that the wire format must not be changed from the message supplied by the application. If we're going to be backwards compatible with the use of ^ (and I think we should and continue to use this) then this already points to the on-wire content of the message being different to what is supplied by an application. I can't see any reason why the on-wire format of the message needs to be the same as that supplied by the application. It's not like we use tcpdump or ethereal to do our loging. Darren ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
Re: [Syslog] #2, max message size
> Darren, > > > The only place a message size limit should be specified is in > > a transport > > mapping. If it's in -15 then it should be removed. Limits > > of all sizes > > and types do nothing but contribute to aging of a protocol. > > -protocol-15 is a compromise after a very long discussion. It says: > > - >A receiver MUST be able to accept messages up to and including 480 >octets in length. For interoperability reasons, all receiver >implementations SHOULD be able to accept messages up to and including >2,048 octets in length. > >If a receiver receives a message with a length larger than 2,048 >octets, or larger than it supports, the receiver MAY discard the >message or truncate the payload. > - > > I think this text is useful. It keeps the door open for any size > messages while still allowing it to be restricted by the transport > mappings and individual implementations (e.g. on low-end embedded > devices). It cautions implementors against being too verbose but also > sets a lower limit that each implementation can assume to be received. > > I think we should continue to use this text. Do you agree? No. That text doesn't belong in this draft. Darren ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
Re: [Syslog] #3 NUL octets, #4 binary data, #8 octet-counting
> Andrew, > > > Also, if we are to transport syslog over TCP at some stage, > > we need to keep > > a delimiter character free from use in the message. Again, a > > LF would be a > > good choice for this delimiter. > > Here, I disagree. I think we can not set aside a character for this. If > we go for TCP, let's do octet-couting. Its reliable, efficient and > proven. Anyhow, we are not yet doing a TCP mapping, so I suggest we save > this discussion until later. Why not use LF? It will work. There are syslogd implementations about that use LF as a record delimeter. In other emails you're saying "we must support binary because some people are going to use this". Now we've got people that support LF as a record delimeter already but you don't want to take this into account. Why does what one vendor is going to do mean more than what other vendors are already doing ? Is there bias here somewhere ? Do you have any particular preference based on work you've done or through some sort of affiliation ? If you want to discount use of LF as a record delimeter then there is no reason we must support use of binary because what vendors are doing is obviously irrelevant. Darren ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
RE: [Syslog] #3 NUL octets, #4 binary data, #8 octet-counting
Hi, Let's keep in mind that syslog-sign will transport binary signatures. In that, the authors are proposing to use base-64 encoding. I agree with Rainer that we've provided enough means in syslog-protocol so that may be accomplished. As Rainer says, let's focus on getting syslog-protocol finished. :) Thanks, Chris On Wed, 30 Nov 2005, Rainer Gerhards wrote: Andrew, That sounds good to me at this stage, and it keeps the door open. I would prefer to see all binary data encoded in some "safe" format like base64. It just makes logging the data to file much easier. For instance, if the binary data contained a LF character, when it was logged to disk, it would end up as two separate messages when opened in notepad etc. While I too prefer a "safe" format, I would not like to outrule another one. Maybe later... (we *need* to finish this I-D...). How it is handled when writing to disk is beyond IETF. I would escape it then - but that's up to the implementation. Also, if we are to transport syslog over TCP at some stage, we need to keep a delimiter character free from use in the message. Again, a LF would be a good choice for this delimiter. Here, I disagree. I think we can not set aside a character for this. If we go for TCP, let's do octet-couting. Its reliable, efficient and proven. Anyhow, we are not yet doing a TCP mapping, so I suggest we save this discussion until later. Rainer PS: sorry for trying to focus on the immediate needs. But I think we really need to finish something... Cheers Andrew -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Rainer Gerhards Sent: Wednesday, 30 November 2005 9:26 p.m. To: [EMAIL PROTECTED] Subject: [Syslog] #3 NUL octets, #4 binary data, #8 octet-counting Hi WG, I have received notes via private mail telling me there seem to be some existing (and eventually soon upcoming) valid use cases for binary data in syslog. I think there is no point in arguing whether that's fortunate or not. It simply looks like that's the way it is. I do not like the idea of breaking existing use cases for syslog (because that will only lead to implementors ignoring the spec and the story of syslog inconsistencies continues...). As such, I think we need to provide at least some minimal support for it (aka "not outlaw it"). At first, this implies that NUL octets may be present in the message. I propose that we write text that discourages the use of NUL, but allows it if needed. That text should also allow, but discourage, a receiver to modify messages containing NUL. With that, we allow the use case, but do not make it a "show stopper" for implementing compliant software. This would also be pretty much in sync with what we currently find in practice, so it is already expected behaviour. Finally, such text would caution implementors that when NUL octets are present, chancs are high that eventually present digitial signatures will be broken. In my point of view, that's fair and efficient. Chris proposal for #5 (character encoding) also provides an elegant solution for binary data. We can use something like: [enc="binary"] or [enc="base-64"] I do NOT intend to specify this - I think it should be in the scope of a separate document specifying the use of binary data. Then would also be the right time to discuss all issues that arise out of it. For now, I just would like to keep the door open. Finally, I propose to extend Chris format so that the message size can be conveyed. This has been brought up several times and I think a clean solution is now obvious: [enc="utf-8" lang="en" size="MSG-size-in-octets"] MSG-size-in-octets would be the size of the MSG part (just that!) in octets. Counting just the MSG part is sufficient, as the rest of the message consists of fields properly delimited. The size is probably most useful for binary data. Please comment. Rainer ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
RE: [Syslog] #5 - character encoding (was: Consensus?)
Chris, I fully agree - thanks ;) Rainer > -Original Message- > From: Chris Lonvick [mailto:[EMAIL PROTECTED] > Sent: Wednesday, November 30, 2005 2:39 PM > To: Rainer Gerhards > Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED] > Subject: RE: [Syslog] #5 - character encoding (was: Consensus?) > > Hi Rainer, > > I believe that we are saying the same thing. :) > > If there is no indicator of encoding or language then a > reciever will not > know what it is receiving - just like receivers don't know > what they are > receiving today. They MAY make an assumption that it is something in > US-ASCII (but may be disappointed). > > If there is an indicator of the encoding and language then > the receiver > will know exactly what it is. Having an indicator should be > RECOMMENDED > but not REQUIRED for ease of migration. > > Is that what we're all saying? > > Thanks, > Chris > > > > On Wed, 30 Nov 2005, Rainer Gerhards wrote: > > > Chris, > > > >> Let's use this email as an example. :) There is no > >> indication that I'm > >> using US-ASCII encoding or that I'm writing in English. > > > > I think there actually is. If I am right, the SMTP RFCs > require mail text to be US-ASCII. Only via MIME and/or escape > characters you can include 8-bit data. For example Müller and > Möller might create some problems in some mailers (But I > guess my Mail system will encode them with =). > Dropping messages with octets > 127 in the subject is a > common spam protection setting... > > > >> However, you're > >> able to recieve this and read it. Similarly, you could write > >> an email in > >> German and send it to me. I would still be able to recieve > >> it but I'd > >> have a difficult time parsing the meaning. > >> > >> I'm suggesting that same approach for the transmission of > the syslog > >> content. If I really wanted you to know what encoding and > >> language I'm > >> using in an email, I would specify a mime header. syslog > >> senders will > >> continue to pump out whatever encoding and language they've > >> been using > >> and recievers will continue to do their best to parse them. > >> If a vendor > >> wants to get very specific about that, then they will have to > >> use an SD-ID > >> to identify the contents of the message. > > > > Here I agree with you. What I was saying is that IF the > header says it is US-ASCII, only then we should assume it > actually is. If there is no "enc" SD-ID, then we do not know > what it is but can assume ... whatever we assume. Let me > phrase it that way: > > > > If the message contains > > > > [enc="us-ascii" lang="en"] > > > > then the receiver can honestly expect it to be US-ASCII. > But if it does not contain any "enc" the receiver does not > know exactly and assume anything it finds useful (may be > ASCII, may not). > > > > Does this clarify? I somehow have the impression we mean > the same thing and I simply do not manage to convey what I > intend to ;) > > > > Rainer > > > >> > >> Mit Aufrichtigkeit, > >> Chris > >> > >> > >> > >> > >> On Wed, 30 Nov 2005, Rainer Gerhards wrote: > >> > >>> Andrew, > >>> > > Hi Rainer, > > > > Why don't we look at it from the other direction? We could > state that any > > encoding is acceptable - for ease-of-use/migration with > existing syslog > > implementations. It is RECOMMENDED that UTF-8 be used. > >> When it is > > used, an SD-ID element will be REQUIRED. e.g. - > [enc="utf-8" lang="en"] > > I like that idea too. > > So, if no SD-ID encoding element is specified, then we must > assume US-ASCII > and deal with it accordingly?? > >>> > >>> I think not. If it is not present, we known that we do not > >> know it. If > >>> it is US-ASCII, I would expect something like > >>> > >>> [enc="us-ascii" lang="en"] > >>> > >>> Of course, we could also say if it is non-present, we can assume > >>> US-ASCII. But then we would need to introduce > >>> > >>> [enc="unknown"] > >>> > >>> for the (common) case where we simply do not know it (again: think > >>> POSIX). I find this somehwat confusing. > >>> > >>> Rainer > >>> > >> > > > ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
RE: [Syslog] #5 - character encoding (was: Consensus?)
Hi Rainer, I believe that we are saying the same thing. :) If there is no indicator of encoding or language then a reciever will not know what it is receiving - just like receivers don't know what they are receiving today. They MAY make an assumption that it is something in US-ASCII (but may be disappointed). If there is an indicator of the encoding and language then the receiver will know exactly what it is. Having an indicator should be RECOMMENDED but not REQUIRED for ease of migration. Is that what we're all saying? Thanks, Chris On Wed, 30 Nov 2005, Rainer Gerhards wrote: Chris, Let's use this email as an example. :) There is no indication that I'm using US-ASCII encoding or that I'm writing in English. I think there actually is. If I am right, the SMTP RFCs require mail text to be US-ASCII. Only via MIME and/or escape characters you can include 8-bit data. For example Müller and Möller might create some problems in some mailers (But I guess my Mail system will encode them with =). Dropping messages with octets > 127 in the subject is a common spam protection setting... However, you're able to recieve this and read it. Similarly, you could write an email in German and send it to me. I would still be able to recieve it but I'd have a difficult time parsing the meaning. I'm suggesting that same approach for the transmission of the syslog content. If I really wanted you to know what encoding and language I'm using in an email, I would specify a mime header. syslog senders will continue to pump out whatever encoding and language they've been using and recievers will continue to do their best to parse them. If a vendor wants to get very specific about that, then they will have to use an SD-ID to identify the contents of the message. Here I agree with you. What I was saying is that IF the header says it is US-ASCII, only then we should assume it actually is. If there is no "enc" SD-ID, then we do not know what it is but can assume ... whatever we assume. Let me phrase it that way: If the message contains [enc="us-ascii" lang="en"] then the receiver can honestly expect it to be US-ASCII. But if it does not contain any "enc" the receiver does not know exactly and assume anything it finds useful (may be ASCII, may not). Does this clarify? I somehow have the impression we mean the same thing and I simply do not manage to convey what I intend to ;) Rainer Mit Aufrichtigkeit, Chris On Wed, 30 Nov 2005, Rainer Gerhards wrote: Andrew, Hi Rainer, Why don't we look at it from the other direction? We could state that any encoding is acceptable - for ease-of-use/migration with existing syslog implementations. It is RECOMMENDED that UTF-8 be used. When it is used, an SD-ID element will be REQUIRED. e.g. - [enc="utf-8" lang="en"] I like that idea too. So, if no SD-ID encoding element is specified, then we must assume US-ASCII and deal with it accordingly?? I think not. If it is not present, we known that we do not know it. If it is US-ASCII, I would expect something like [enc="us-ascii" lang="en"] Of course, we could also say if it is non-present, we can assume US-ASCII. But then we would need to introduce [enc="unknown"] for the (common) case where we simply do not know it (again: think POSIX). I find this somehwat confusing. Rainer ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
RE: [Syslog] #5 - character encoding (was: Consensus?)
Chris, > Let's use this email as an example. :) There is no > indication that I'm > using US-ASCII encoding or that I'm writing in English. I think there actually is. If I am right, the SMTP RFCs require mail text to be US-ASCII. Only via MIME and/or escape characters you can include 8-bit data. For example Müller and Möller might create some problems in some mailers (But I guess my Mail system will encode them with =). Dropping messages with octets > 127 in the subject is a common spam protection setting... > However, you're > able to recieve this and read it. Similarly, you could write > an email in > German and send it to me. I would still be able to recieve > it but I'd > have a difficult time parsing the meaning. > > I'm suggesting that same approach for the transmission of the syslog > content. If I really wanted you to know what encoding and > language I'm > using in an email, I would specify a mime header. syslog > senders will > continue to pump out whatever encoding and language they've > been using > and recievers will continue to do their best to parse them. > If a vendor > wants to get very specific about that, then they will have to > use an SD-ID > to identify the contents of the message. Here I agree with you. What I was saying is that IF the header says it is US-ASCII, only then we should assume it actually is. If there is no "enc" SD-ID, then we do not know what it is but can assume ... whatever we assume. Let me phrase it that way: If the message contains [enc="us-ascii" lang="en"] then the receiver can honestly expect it to be US-ASCII. But if it does not contain any "enc" the receiver does not know exactly and assume anything it finds useful (may be ASCII, may not). Does this clarify? I somehow have the impression we mean the same thing and I simply do not manage to convey what I intend to ;) Rainer > > Mit Aufrichtigkeit, > Chris > > > > > On Wed, 30 Nov 2005, Rainer Gerhards wrote: > > > Andrew, > > > >>> Hi Rainer, > >>> > >>> Why don't we look at it from the other direction? We could > >> state that any > >>> encoding is acceptable - for ease-of-use/migration with > >> existing syslog > >>> implementations. It is RECOMMENDED that UTF-8 be used. > When it is > >>> used, an SD-ID element will be REQUIRED. e.g. - > >> [enc="utf-8" lang="en"] > >> > >> I like that idea too. > >> > >> So, if no SD-ID encoding element is specified, then we must > >> assume US-ASCII > >> and deal with it accordingly?? > > > > I think not. If it is not present, we known that we do not > know it. If > > it is US-ASCII, I would expect something like > > > > [enc="us-ascii" lang="en"] > > > > Of course, we could also say if it is non-present, we can assume > > US-ASCII. But then we would need to introduce > > > > [enc="unknown"] > > > > for the (common) case where we simply do not know it (again: think > > POSIX). I find this somehwat confusing. > > > > Rainer > > > ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
RE: [Syslog] #5 - character encoding (was: Consensus?)
Chris, I agree to all but one point - only that one quoted here... > > Also want to clarify that you suggest that if the message > is in ASCII, > > it will not required SD-ID, but for all other encodings, > SD-ID will be > > required. > > Yes - that's my suggestion. I am sorry, we can not do this. The whole issue is rooted in POSIX APIs. You need to look at it why it is such a problem. On Windows, you know what character encodings you are dealing with. On Unix, you actually just get a bunch of octets - and nobody tells you what it is. So the poor Unix syslogd actually has no idea of what it handles and likewise does not know what to place in that field ;) If it knew it were this or that encoding, I would be very tempted to request it to convert to UTF-8. But the need behind this encoding is *NOT* to allow the multitude of whatever currently is in existence but rather provide a way to let a syslogd that needs to omit a "bunch of octets" do that. Does this clarify? I can provide code if that would be helpful... Rainer ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
RE: [Syslog] #3 NUL octets, #4 binary data, #8 octet-counting
Andrew, > That sounds good to me at this stage, and it keeps the door > open. I would > prefer to see all binary data encoded in some "safe" format > like base64. It > just makes logging the data to file much easier. For > instance, if the binary > data contained a LF character, when it was logged to disk, it > would end up > as two separate messages when opened in notepad etc. While I too prefer a "safe" format, I would not like to outrule another one. Maybe later... (we *need* to finish this I-D...). How it is handled when writing to disk is beyond IETF. I would escape it then - but that's up to the implementation. > Also, if we are to transport syslog over TCP at some stage, > we need to keep > a delimiter character free from use in the message. Again, a > LF would be a > good choice for this delimiter. Here, I disagree. I think we can not set aside a character for this. If we go for TCP, let's do octet-couting. Its reliable, efficient and proven. Anyhow, we are not yet doing a TCP mapping, so I suggest we save this discussion until later. Rainer PS: sorry for trying to focus on the immediate needs. But I think we really need to finish something... > Cheers > > Andrew > > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] > On Behalf Of Rainer Gerhards > Sent: Wednesday, 30 November 2005 9:26 p.m. > To: [EMAIL PROTECTED] > Subject: [Syslog] #3 NUL octets, #4 binary data, #8 octet-counting > > > Hi WG, > > I have received notes via private mail telling me there seem > to be some > existing (and eventually soon upcoming) valid use cases for > binary data > in syslog. I think there is no point in arguing whether > that's fortunate > or not. It simply looks like that's the way it is. I do not like the > idea of breaking existing use cases for syslog (because that will only > lead to implementors ignoring the spec and the story of syslog > inconsistencies continues...). As such, I think we need to provide at > least some minimal support for it (aka "not outlaw it"). > > At first, this implies that NUL octets may be present in the message. > > I propose that we write text that discourages the use of NUL, > but allows > it if needed. That text should also allow, but discourage, a > receiver to > modify messages containing NUL. With that, we allow the use > case, but do > not make it a "show stopper" for implementing compliant software. This > would also be pretty much in sync with what we currently find in > practice, so it is already expected behaviour. Finally, such > text would > caution implementors that when NUL octets are present, chancs are high > that eventually present digitial signatures will be broken. > In my point > of view, that's fair and efficient. > > Chris proposal for #5 (character encoding) also provides an elegant > solution for binary data. We can use something like: > > [enc="binary"] > > or > > [enc="base-64"] > > I do NOT intend to specify this - I think it should be in the > scope of a > separate document specifying the use of binary data. Then > would also be > the right time to discuss all issues that arise out of it. For now, I > just would like to keep the door open. > > Finally, I propose to extend Chris format so that the message size can > be conveyed. This has been brought up several times and I > think a clean > solution is now obvious: > > [enc="utf-8" lang="en" size="MSG-size-in-octets"] > > MSG-size-in-octets would be the size of the MSG part (just that!) in > octets. Counting just the MSG part is sufficient, as the rest of the > message consists of fields properly delimited. The size is > probably most > useful for binary data. > > Please comment. > > Rainer > > ___ > Syslog mailing list > Syslog@lists.ietf.org > https://www1.ietf.org/mailman/listinfo/syslog > > ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
RE: [Syslog] #5 - character encoding (was: Consensus?)
Hi Rainer, Let's use this email as an example. :) There is no indication that I'm using US-ASCII encoding or that I'm writing in English. However, you're able to recieve this and read it. Similarly, you could write an email in German and send it to me. I would still be able to recieve it but I'd have a difficult time parsing the meaning. I'm suggesting that same approach for the transmission of the syslog content. If I really wanted you to know what encoding and language I'm using in an email, I would specify a mime header. syslog senders will continue to pump out whatever encoding and language they've been using and recievers will continue to do their best to parse them. If a vendor wants to get very specific about that, then they will have to use an SD-ID to identify the contents of the message. Mit Aufrichtigkeit, Chris On Wed, 30 Nov 2005, Rainer Gerhards wrote: Andrew, Hi Rainer, Why don't we look at it from the other direction? We could state that any encoding is acceptable - for ease-of-use/migration with existing syslog implementations. It is RECOMMENDED that UTF-8 be used. When it is used, an SD-ID element will be REQUIRED. e.g. - [enc="utf-8" lang="en"] I like that idea too. So, if no SD-ID encoding element is specified, then we must assume US-ASCII and deal with it accordingly?? I think not. If it is not present, we known that we do not know it. If it is US-ASCII, I would expect something like [enc="us-ascii" lang="en"] Of course, we could also say if it is non-present, we can assume US-ASCII. But then we would need to introduce [enc="unknown"] for the (common) case where we simply do not know it (again: think POSIX). I find this somehwat confusing. Rainer ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
RE: [Syslog] #5 - character encoding (was: Consensus?)
Hi Sheran,
On Tue, 29 Nov 2005, Shyyunn Lin (sheranl) wrote:
Chris:
I think having SD-ID with [enc="utf-8" lang="English"] may be a good
approach. If different language use utf-8 encoding, then "lang=" can
distinguish it.
We _should_ be using language codes from RFC 3066. That specifies ISO 639
language tags. 639-1 has 2 character codes ("en" is English) and 639-2
has 3 characters ("eng" is English). RFC 3066 will likely be replaced by
the works of the Language Tag Registry Update (ltru) Working Group.
http://www.ietf.org/html.charters/ltru-charter.html
They have IDs in the works. Until those become RFCs we should continue to
reference RFC 3066.
Also want to clarify that you suggest that if the message is in ASCII,
it will not required SD-ID, but for all other encodings, SD-ID will be
required.
Yes - that's my suggestion.
Note most other encoding methods already imply the language used, for
example, in Chinese, there are several encoding methods, Traditional
Chinese used in Taiwan and Hong Kong is Big5, and simplified Chinese
used in Mainland China is GBK, so if the message is in traditional
Chinese char, it will be shown as [enc="Big5", lang="Traditional
Chinese"], a little bit redundant. The Big5 also includes all English
char so it can be a mix of Chinese and English.
Good point. As far as I can tell, "Big5" is not recognized by any
accredited standards developing organization. It is recognized by the
Ideographic Rapporteur Group (IRG) which reports to the Unicode
consortium. The recognized way to represent Chinese characters,
traditional and simplified, is through ISO 639-2 with the subcodes to
indicate traditional and simplified for the "zh" _language_. The ID on
"Tags for Identifying Languages"
http://www.ietf.org/internet-drafts/draft-ietf-ltru-registry-14.txt
identifies simplified Chinese as "zh-Hans" and traditional Chinese as
"zh-Hant". Additional subtags could identify a locale such as
"zh-Hant-TW" for Taiwan Chinese in traditional script. This is from the
"Initial Language Subtag Registry" ID.
http://www.ietf.org/internet-drafts/draft-ietf-ltru-initial-06.txt
I think that we should specify encoding and language tags as
striaghtforward as possible and let others augment syslog-protocol (in the
future) with other encoding mechanisms. We can RECOMMEND that encoding be
in UTF-8 and language tags come from RFC 3066. We can allow that other
encoding and language identifications are acceptable. In the worst case,
a vendor will have the option of [EMAIL PROTECTED]"something" [EMAIL PROTECTED]"piglatin"].
Does this work for you?
Thanks,
Chris
Regards,
Sheran
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Chris Lonvick
(clonvick)
Sent: Tuesday, November 29, 2005 10:22 AM
To: Rainer Gerhards
Cc: [EMAIL PROTECTED]
Subject: RE: [Syslog] #5 - character encoding (was: Consensus?)
Hi Rainer,
Why don't we look at it from the other direction? We could state that
any encoding is acceptable - for ease-of-use/migration with existing
syslog implementations. It is RECOMMENDED that UTF-8 be used. When it
is used, an SD-ID element will be REQUIRED. e.g. - [enc="utf-8"
lang="en"]
Thoughts?
All: Let's discuss this and close this issue.
Thanks,
Chris
On Tue, 29 Nov 2005, Rainer Gerhards wrote:
Chris & WG,
#5 Character encoding in MSG: due to my proof-of-concept
implementation, I have raised the (ugly) question if we need
to allow encodings other than UTF-8. Please note that this
question arises from needs introduced by e.g. POSIX. So we
can't easily argue them away by whishful thinking ;)
Not even discussed yet.
I haven't reviewed that yet. However, I'll note that allowing
different encoding can be accomplished in the future as long as we
establish a default encoding and a way to identify it in our current
work.
I have read a little in the mailing archive. Please note that in 2000
it was consensus that the MSG part may contain encodings other then
US-ASCII. Follow this threat:
http://www.syslog.cc/ietf/autoarc/msg00127.html
This discussion lead to RFC 3164 saying "other encodings MAY be used".
While this was observed behaviour, we need still to be aware that the
POSIX (and glibc) API places the restrictions on us that we simply do
not know the character encoding used by the application. As such, no
*nix syslogd can be programmed to be compliant to syslog-protocol if
we demand UTF-8 exclusively.
I propose that we RECOMMEND UTF-8 that MUST start with the Unicode
Byte Order Mask (BOM) if used. If the MSG part does not start with the
BOM, it may be any encoding just as in RFC 3164. I do not see any
alternative to this.
Rainer
___
Syslog mailing list
Syslog@lists.ietf.org
https://www1.ietf.org/mailman/listinfo/syslog
___
Syslog mailing list
Syslog@lists.ietf.org
https://www1.ietf.org/mailma
RE: [Syslog] #3 NUL octets, #4 binary data, #8 octet-counting
Rainer, That sounds good to me at this stage, and it keeps the door open. I would prefer to see all binary data encoded in some "safe" format like base64. It just makes logging the data to file much easier. For instance, if the binary data contained a LF character, when it was logged to disk, it would end up as two separate messages when opened in notepad etc. Also, if we are to transport syslog over TCP at some stage, we need to keep a delimiter character free from use in the message. Again, a LF would be a good choice for this delimiter. Cheers Andrew -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Rainer Gerhards Sent: Wednesday, 30 November 2005 9:26 p.m. To: [EMAIL PROTECTED] Subject: [Syslog] #3 NUL octets, #4 binary data, #8 octet-counting Hi WG, I have received notes via private mail telling me there seem to be some existing (and eventually soon upcoming) valid use cases for binary data in syslog. I think there is no point in arguing whether that's fortunate or not. It simply looks like that's the way it is. I do not like the idea of breaking existing use cases for syslog (because that will only lead to implementors ignoring the spec and the story of syslog inconsistencies continues...). As such, I think we need to provide at least some minimal support for it (aka "not outlaw it"). At first, this implies that NUL octets may be present in the message. I propose that we write text that discourages the use of NUL, but allows it if needed. That text should also allow, but discourage, a receiver to modify messages containing NUL. With that, we allow the use case, but do not make it a "show stopper" for implementing compliant software. This would also be pretty much in sync with what we currently find in practice, so it is already expected behaviour. Finally, such text would caution implementors that when NUL octets are present, chancs are high that eventually present digitial signatures will be broken. In my point of view, that's fair and efficient. Chris proposal for #5 (character encoding) also provides an elegant solution for binary data. We can use something like: [enc="binary"] or [enc="base-64"] I do NOT intend to specify this - I think it should be in the scope of a separate document specifying the use of binary data. Then would also be the right time to discuss all issues that arise out of it. For now, I just would like to keep the door open. Finally, I propose to extend Chris format so that the message size can be conveyed. This has been brought up several times and I think a clean solution is now obvious: [enc="utf-8" lang="en" size="MSG-size-in-octets"] MSG-size-in-octets would be the size of the MSG part (just that!) in octets. Counting just the MSG part is sufficient, as the rest of the message consists of fields properly delimited. The size is probably most useful for binary data. Please comment. Rainer ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
RE: [Syslog] formal Consultation prior to concluding the working group
Sam, with this, I am updating my formal reply from November, 18th. Since them, positive signs have emerged. I agree with Chris on that. We have had a discussion on a new charter and I think we have consensus and/or are very close to it. Also, a lot of technical discussion has been going on. That shows the ability of the WG to look at technical issues and work towards a solution. If I am not totally wrong, rough consensus can be reached soon. Also, I have done a proof-of-concept implementation in the mean time which revealed some other important issues and also serves as proof for some technical arguments. With the post-Vancouver changes, the implementation was relatively effortless. Lab testing has been done, a thing the WG as whole did not do enough before Vancouver. This lab testing has revealed that compatibility with RFC 3164 is not extremely important (except for PRI, which is vital). More implementors have joined the list and Chris has managed to motiviate some reviewers. Finally, the candidate format for the next revision has shown in lab testing that it provides adequate backwards compatibility. I have summarized all of this on http://www.syslog.cc/ietf/protocol.html#200511 which I invite you to at least quickly review. It also has extended reasoning why RFC 3164 does not tell the "real" truth, a fact that probably is important for any further decisions. So far the good news. On the negative side, the consensus still seems to be very fragile. Also, some issues are re-itereated ever and ever again. While WG list participation has increased, there are currently still only 15 voices heared, 3 of them only once (I have counted posters in the mailing list archive - mails send after the Vancouver meeting). The actual discussion is still limited to few people, but that might be acceptable. As Anton mentioned, there are few experts in syslog logging and the overall interest is low. I have also seen the NETCONF WG discussing a syslog replacement after the Vancouver meeting. I see they struggle with a set of similar questions. There seem to be more participants. If NETCONF redos event logging, the advantage is that the do not need to look at any backwards compatibility at all. Should the syslog WG fail to complete its work, NETCONF event notifications could be the alternative in the long term. However, given the wide-spread use of syslog, I doubt that syslog will ever be totally replaced by something else. In an ideal world, I would like to see a revived syslog WG work together with NETCONF on this issue. To be honest, I have not yet tried to influence the NETCONF discussion because I am unsure about the state of the syslog WG. If I weigh pros and cons, I think that the WG should be allowed to try to complete its work. Especially the fact that lab testing has revealed such large incompatibilities of existing deployed technology, together with the relative ease of fixing it, provides reasoning enough for it being alive. HOWEVER, I also think we need to deliver documents, not just arguments. So my proposal is that the WG be rechartered based on the soon-to-be reached consensus of the charter. Then, a challenging milestone (no later than the next IETF meeting) should be set for deliverying syslog-protocol and syslog-transport-udp. If that milestone can not be reached, I suggest concluding the WG because of its inability to deliver results. We are discussing this for roughly 3 years now, I and others have spent considerable time in trying to solve it and it does not make sense to continue if we can not reach consensus. What concerns me in this regard is that I still do not know how we can avoid such an "out of the sudden" discrepancy between IETF meeting and list consensus. Not meeting, I think, is not an option. The feedback from Vancouver was important. I just would like to see this on-list, too. Let me close my conclusion with the reminder that all what I have said is my personal view. The WG may or may not agree with this - I haven't even tried to reach consensus on it. I hope it is useful in your decision process, but that's all it is about. I personally commit to try to finish syslog-protocol, but I will definitely drop out of this effort mid-next year at latest, no matter what the overall route then might be. I hope we can finish things quickly and then focus on other small documents. There is lots that could be done and should be done in syslog. Just we need to do small steps, small documents as Darren rightly proposed. We need to iron them out within a few month and not years. I am positive about all this and think Chris has already guided the WG into the right direction. My best regards, Rainer > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Rainer Gerhards > Sent: Friday, November 18, 2005 10:01 AM > To: [EMAIL PROTECTED]; [EMAIL PROTECTED] > Cc: [EMAIL PROTECTED]; Darren Reed > Subject: RE: [Syslog] formal Consultation prior to conclu
RE: [Syslog] #5 - character encoding (was: Consensus?)
Andrew, > >Hi Rainer, > > > >Why don't we look at it from the other direction? We could > state that any > >encoding is acceptable - for ease-of-use/migration with > existing syslog > >implementations. It is RECOMMENDED that UTF-8 be used. When it is > >used, an SD-ID element will be REQUIRED. e.g. - > [enc="utf-8" lang="en"] > > I like that idea too. > > So, if no SD-ID encoding element is specified, then we must > assume US-ASCII > and deal with it accordingly?? I think not. If it is not present, we known that we do not know it. If it is US-ASCII, I would expect something like [enc="us-ascii" lang="en"] Of course, we could also say if it is non-present, we can assume US-ASCII. But then we would need to introduce [enc="unknown"] for the (common) case where we simply do not know it (again: think POSIX). I find this somehwat confusing. Rainer ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
RE: [Syslog] #5 - character encoding (was: Consensus?)
>Hi Rainer, > >Why don't we look at it from the other direction? We could state that any >encoding is acceptable - for ease-of-use/migration with existing syslog >implementations. It is RECOMMENDED that UTF-8 be used. When it is >used, an SD-ID element will be REQUIRED. e.g. - [enc="utf-8" lang="en"] I like that idea too. So, if no SD-ID encoding element is specified, then we must assume US-ASCII and deal with it accordingly?? Cheers Andrew ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
RE: [Syslog] #3 NUL octets, #4 binary data, #8 octet-counting
Darren, > > I have received notes via private mail telling me there > seem to be some > > existing (and eventually soon upcoming) valid use cases for > binary data > > in syslog. I think there is no point in arguing whether > that's fortunate > > or not. It simply looks like that's the way it is. I do not like the > > idea of breaking existing use cases for syslog (because > that will only > > lead to implementors ignoring the spec and the story of syslog > > inconsistencies continues...). As such, I think we need to > provide at > > least some minimal support for it (aka "not outlaw it"). > .. > > This is the wrong approach to take. > > Furthermore, if 3164 is anything to go by, conveying of > control characters > by the existing protocol is not well documented at present. That is right, RFC 3164 does NOT well document existing behaviour. Maybe that's why it is titled "the BSD syslog protocol" - just kidding. I have made considerable effort the past days to see why there is such a bad mapping between RFC 3164 and reality. In fact, I consider this to be a major problem that surfaces often during discussions. I myself based lots of assumptions on RFC 3164 and now testing has shown that these are invalid. If I only had gone to the lab earlier... I've read through the full mailing list archive yesterday. I have seen that there was put a lot of effort in RFC 3164, Chris has even talked several times with Eric Allman, who unquestionable knows pretty well about syslog ;) So why this gap between 3164 and practice? I have come to the conclusion that after Eric invented it, other folks have redone his work on other platforms (e.g. Linux, Windows and of course embedded devices). While all of these implementors had Eric's ideas on their mind, there was no spec to follow. Thus, each one introduced subtle differences and finally even BSD syslogd was modified in subtle ways. At the time 3164 was defined, nobody went to the lab and verified the results. Nor did I when I started with syslog-protocol. We all were so convinced that when Eric agreed to the document, it must be right. I think we do not need to put finger at anyone. In fact, 3164 is a fine piece of work. Even if the format specification can not be considered to be universally, it describes the security issues inherent with syslog. As far as I understand, that was the primary goal of that document. But now that we know practice is different, we should apply that knowledge to new documents. > > I think it is incumbant upon the WG to convey the message to > implementors > that while it will listen to their needs and plans for > implementation, it > is not up to them to choose which way this group proceeds > merely because > they decide to provide a particular implementation. In general, I agree. But I think we should support the currently existing use cases for syslog. After all, was this effort not put on hold because it was said it is not backwards-compatible enough? I do not intend to encourage that, that's also why I suggest wording on the restrictions (like no signatures) it comes with. > > .. > > Chris proposal for #5 (character encoding) also provides an elegant > > solution for binary data. We can use something like: > > > > [enc="binary"] > > > > or > > > > [enc="base-64"] > > > > I do NOT intend to specify this - I think it should be in > the scope of a > > separate document specifying the use of binary data. Then > would also be > > the right time to discuss all issues that arise out of it. > For now, I > > just would like to keep the door open. > > > > Finally, I propose to extend Chris format so that the > message size can > > be conveyed. This has been brought up several times and I > think a clean > > solution is now obvious: > > > > [enc="utf-8" lang="en" size="MSG-size-in-octets"] > > > > MSG-size-in-octets would be the size of the MSG part (just that!) in > > octets. Counting just the MSG part is sufficient, as the rest of the > > message consists of fields properly delimited. The size is > probably most > > useful for binary data. > > What happens if the message is truncated? Good question. I'd say the size would need to be adjusted. > What value is the message size really providing here? > If we have a natural EOR marker, LF, what do we need the size for? We do NOT have a EOR marker. As of the current draft, LF is just a regular character like any other. We explicitely wanted the capability to surface, now we have it. I even think its not bad to have it. I do NOT like the idea to re-iterate that long discussion of whether or not full UTF-8 should be supported. Please review the mailing list archive for that. If you violently object, please do so. In this case, I would ask Chris how we can proceed in such a case. I myself violently object restricting to printable characters only because of the multiple reasons found in the archive (and I don't intend to re-iterate that for the 6th time...). Sorry to be blu
RE: [Syslog] #7 field order
I just got private mail if a missing field is denoted by "-". This is the case. Optional fields should be all but VERSION. Rainer > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Rainer Gerhards > Sent: Wednesday, November 30, 2005 9:37 AM > To: [EMAIL PROTECTED] > Subject: [Syslog] #7 field order > > WG, > > there has not been much discussion about the header fields and their > order recently. I think this is a sign the issue has been settled. To > make sure I got the right understanding of the resulting consensus, I > propose that we use the following format: > > VERSION SP TIMESTAMP SP HOSTNAME SP APP-NAME SP PROCID > SP MSGID SP > [SD-ID]s SP MSG > > That is the format that also proven to be quite useful during my > proof-of-concept implementation. > > If somebody objects, please do that now. > > Thanks, > Rainer > > ___ > Syslog mailing list > Syslog@lists.ietf.org > https://www1.ietf.org/mailman/listinfo/syslog > ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
Re: [Syslog] #3 NUL octets, #4 binary data, #8 octet-counting
> Hi WG, > > I have received notes via private mail telling me there seem to be some > existing (and eventually soon upcoming) valid use cases for binary data > in syslog. I think there is no point in arguing whether that's fortunate > or not. It simply looks like that's the way it is. I do not like the > idea of breaking existing use cases for syslog (because that will only > lead to implementors ignoring the spec and the story of syslog > inconsistencies continues...). As such, I think we need to provide at > least some minimal support for it (aka "not outlaw it"). .. This is the wrong approach to take. Furthermore, if 3164 is anything to go by, conveying of control characters by the existing protocol is not well documented at present. I think it is incumbant upon the WG to convey the message to implementors that while it will listen to their needs and plans for implementation, it is not up to them to choose which way this group proceeds merely because they decide to provide a particular implementation. .. > Chris proposal for #5 (character encoding) also provides an elegant > solution for binary data. We can use something like: > > [enc="binary"] > > or > > [enc="base-64"] > > I do NOT intend to specify this - I think it should be in the scope of a > separate document specifying the use of binary data. Then would also be > the right time to discuss all issues that arise out of it. For now, I > just would like to keep the door open. > > Finally, I propose to extend Chris format so that the message size can > be conveyed. This has been brought up several times and I think a clean > solution is now obvious: > > [enc="utf-8" lang="en" size="MSG-size-in-octets"] > > MSG-size-in-octets would be the size of the MSG part (just that!) in > octets. Counting just the MSG part is sufficient, as the rest of the > message consists of fields properly delimited. The size is probably most > useful for binary data. What happens if the message is truncated? What value is the message size really providing here? If we have a natural EOR marker, LF, what do we need the size for? Given that a syslog message is a single record, I don't believe that it makes any sense to include a "size" parameter. Darren ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
RE: [Syslog] #2, max message size
Darren, > The only place a message size limit should be specified is in > a transport > mapping. If it's in -15 then it should be removed. Limits > of all sizes > and types do nothing but contribute to aging of a protocol. -protocol-15 is a compromise after a very long discussion. It says: - A receiver MUST be able to accept messages up to and including 480 octets in length. For interoperability reasons, all receiver implementations SHOULD be able to accept messages up to and including 2,048 octets in length. If a receiver receives a message with a length larger than 2,048 octets, or larger than it supports, the receiver MAY discard the message or truncate the payload. - I think this text is useful. It keeps the door open for any size messages while still allowing it to be restricted by the transport mappings and individual implementations (e.g. on low-end embedded devices). It cautions implementors against being too verbose but also sets a lower limit that each implementation can assume to be received. I think we should continue to use this text. Do you agree? Rainer ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
Re: [Syslog] #2, max message size
> Hi WG, > > I have not heared back on this topic. So I conclude that it is consensus > that the message size specification be left as it currently is in > syslog-protocol-15. We might put further limits in the transport > mappings. > If someone objects to this view, please do it now as I plan to update > the I-D in the not so distant future. The only place a message size limit should be specified is in a transport mapping. If it's in -15 then it should be removed. Limits of all sizes and types do nothing but contribute to aging of a protocol. Darren ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
[Syslog] #7 field order
WG, there has not been much discussion about the header fields and their order recently. I think this is a sign the issue has been settled. To make sure I got the right understanding of the resulting consensus, I propose that we use the following format: VERSION SP TIMESTAMP SP HOSTNAME SP APP-NAME SP PROCID SP MSGID SP [SD-ID]s SP MSG That is the format that also proven to be quite useful during my proof-of-concept implementation. If somebody objects, please do that now. Thanks, Rainer ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
[Syslog] #9, learnings from proof-of-concept
WG, one discussion topic were the minor things I discovered during my proof-of-concept implementation. If there is no objection, I will address them in the next update of the I-D. So we could discuss them once that is out. The reason is that I want to save some effort by not posting each and every of them as seperate topics. Of course, if you would like to comment now, that would be even more appreciated. If so, please have a look at http://www.rsyslog.com/index.php?module=Static_Docs&func=view&f=/syslog- protocol.html Thanks, Rainer ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
[Syslog] #3 NUL octets, #4 binary data, #8 octet-counting
Hi WG, I have received notes via private mail telling me there seem to be some existing (and eventually soon upcoming) valid use cases for binary data in syslog. I think there is no point in arguing whether that's fortunate or not. It simply looks like that's the way it is. I do not like the idea of breaking existing use cases for syslog (because that will only lead to implementors ignoring the spec and the story of syslog inconsistencies continues...). As such, I think we need to provide at least some minimal support for it (aka "not outlaw it"). At first, this implies that NUL octets may be present in the message. I propose that we write text that discourages the use of NUL, but allows it if needed. That text should also allow, but discourage, a receiver to modify messages containing NUL. With that, we allow the use case, but do not make it a "show stopper" for implementing compliant software. This would also be pretty much in sync with what we currently find in practice, so it is already expected behaviour. Finally, such text would caution implementors that when NUL octets are present, chancs are high that eventually present digitial signatures will be broken. In my point of view, that's fair and efficient. Chris proposal for #5 (character encoding) also provides an elegant solution for binary data. We can use something like: [enc="binary"] or [enc="base-64"] I do NOT intend to specify this - I think it should be in the scope of a separate document specifying the use of binary data. Then would also be the right time to discuss all issues that arise out of it. For now, I just would like to keep the door open. Finally, I propose to extend Chris format so that the message size can be conveyed. This has been brought up several times and I think a clean solution is now obvious: [enc="utf-8" lang="en" size="MSG-size-in-octets"] MSG-size-in-octets would be the size of the MSG part (just that!) in octets. Counting just the MSG part is sufficient, as the rest of the message consists of fields properly delimited. The size is probably most useful for binary data. Please comment. Rainer ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
[Syslog] #2, max message size
Hi WG, I have not heared back on this topic. So I conclude that it is consensus that the message size specification be left as it currently is in syslog-protocol-15. We might put further limits in the transport mappings. If someone objects to this view, please do it now as I plan to update the I-D in the not so distant future. Thanks, Rainer ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
RE: [Syslog] #5 - character encoding (was: Consensus?)
Sheran, > Also want to clarify that you suggest that if the message is in ASCII, > it will not required SD-ID, but for all other encodings, SD-ID will be > required. Unfortunately, we can not do this. If we would know the encoding, we could translate it to UTF-8, as so far is required by syslog-protocol. However, we often do not know which encoding it is. The reason is that the POSIX syslog API does not tell us. So if we want to support POSIX (which I think we must), we must allow a syslog sender to send messages without telling the encoding - simply because it has no way to obtain that knowledge. A syslog sender embedded e.g. in a device does probably not have this restriction. So it SHOULD encode in UTF-8. That will ensure the receiver can understand it. If the sender has absolutely no idea of how to do that, but knows the encoding, then (and only then) it SHOULD specify the encoding. Rainer > > Note most other encoding methods already imply the language used, for > example, in Chinese, there are several encoding methods, Traditional > Chinese used in Taiwan and Hong Kong is Big5, and simplified Chinese > used in Mainland China is GBK, so if the message is in traditional > Chinese char, it will be shown as [enc="Big5", lang="Traditional > Chinese"], a little bit redundant. The Big5 also includes all English > char so it can be a mix of Chinese and English. > > > > Regards, > > Sheran > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Chris Lonvick > (clonvick) > Sent: Tuesday, November 29, 2005 10:22 AM > To: Rainer Gerhards > Cc: [EMAIL PROTECTED] > Subject: RE: [Syslog] #5 - character encoding (was: Consensus?) > > Hi Rainer, > > Why don't we look at it from the other direction? We could state that > any encoding is acceptable - for ease-of-use/migration with existing > syslog implementations. It is RECOMMENDED that UTF-8 be > used. When it > is used, an SD-ID element will be REQUIRED. e.g. - [enc="utf-8" > lang="en"] > > Thoughts? > > All: Let's discuss this and close this issue. > > Thanks, > Chris > > On Tue, 29 Nov 2005, Rainer Gerhards wrote: > > > Chris & WG, > > > >>> #5 Character encoding in MSG: due to my proof-of-concept > >>> implementation, I have raised the (ugly) question if we need > >>> to allow encodings other than UTF-8. Please note that this > >>> question arises from needs introduced by e.g. POSIX. So we > >>> can't easily argue them away by whishful thinking ;) > >>> > >>> Not even discussed yet. > >> > >> I haven't reviewed that yet. However, I'll note that allowing > >> different encoding can be accomplished in the future as long as we > >> establish a default encoding and a way to identify it in > our current > >> work. > > > > I have read a little in the mailing archive. Please note > that in 2000 > > it was consensus that the MSG part may contain encodings other then > > US-ASCII. Follow this threat: > > > > http://www.syslog.cc/ietf/autoarc/msg00127.html > > > > This discussion lead to RFC 3164 saying "other encodings > MAY be used". > > While this was observed behaviour, we need still to be > aware that the > > POSIX (and glibc) API places the restrictions on us that we > simply do > > not know the character encoding used by the application. As > such, no > > *nix syslogd can be programmed to be compliant to > syslog-protocol if > > we demand UTF-8 exclusively. > > > > I propose that we RECOMMEND UTF-8 that MUST start with the Unicode > > Byte Order Mask (BOM) if used. If the MSG part does not > start with the > > > BOM, it may be any encoding just as in RFC 3164. I do not see any > > alternative to this. > > > > Rainer > > > > ___ > > Syslog mailing list > > Syslog@lists.ietf.org > > https://www1.ietf.org/mailman/listinfo/syslog > > > > ___ > Syslog mailing list > Syslog@lists.ietf.org > https://www1.ietf.org/mailman/listinfo/syslog > > ___ > Syslog mailing list > Syslog@lists.ietf.org > https://www1.ietf.org/mailman/listinfo/syslog > ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
