Re: Use of LWSP in ABNF -- consensus call
Since I composed this I saw additional opinions - one for doing nothing, and a couple that I interpreted as something stronger than a warning (e.g. do not use in the future). I still believe there to be rough consensus for a warning. If anybody can suggest (or repost) very specific text this could help the authors. Thanks, Lisa On May 22, 2007, at 4:29 PM, Lisa Dusseault wrote: Thanks for everybody's input on this. I interpret the discussion as showing consensus for a comment with a warning near the definition of LWSP. Details: I counted 18 opinions. I couldn't see anybody arguing for no comment or text whatsoever. I saw arguments against treating this as a Security Consideration. I saw opinions in favour of deprecating the construct, but I am not sure if that's an opinion for or against the health warning (since the definition of deprecation is loose here). In any case, even if you count those as votes against , I still see rough consensus. Lisa The IESG reviewed http://www.ietf.org/internet-drafts/draft- crocker-rfc4234bis-00.txt for publication as Internet Standard and would like to know if there is consensus to recommend against the use of LWSP in future specifications, as it has caused problems recently in DKIM and could cause problems in other places. Some discussion on this point already: - http://www1.ietf.org/mail-archive/web/ietf/current/msg46048.html - http://www1.ietf.org/mail-archive/web/discuss/current/ msg00463.html - http://mipassoc.org/pipermail/ietf-dkim/2007q1/007295.html - https://datatracker.ietf.org/public/pidtracker.cgi? command=view_commentid=66440 (in this tracker comment, Chris Newman recommended to remove LWSP, but for backward-compatibility it's probably better to keep it and recommend against use) Thanks for your input, Lisa Dusseault ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: [ietf-dkim] Re: Use of LWSP in ABNF -- consensus call
Keith Moore said: it could be argued that the best thing to do is to remove ALL of the rules from the ABNF spec, leaving only the language definition and examples. While I don't support this, it does remind me of a problem. I've had various people tell me in the past that ABNF includes Appendix B and, therefore, it is neither necessary to cite the appendix or to define basic concepts yourself. I know that section 1 says that appendix B is separate from its formal status, but I suggest that the introduction to the appendix should make it clear that citing ABNF does *not* include these rules by reference; such inclusion by reference needs to be explicit. -- Clive D.W. Feather | Work: [EMAIL PROTECTED] | Tel:+44 20 8495 6138 Internet Expert | Home: [EMAIL PROTECTED] | Fax:+44 870 051 9937 Demon Internet | WWW: http://www.davros.org | Mobile: +44 7973 377646 THUS plc|| ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: Use of LWSP in ABNF -- consensus call
Thanks for everybody's input on this. I interpret the discussion as showing consensus for a comment with a warning near the definition of LWSP. Details: I counted 18 opinions. I couldn't see anybody arguing for no comment or text whatsoever. I saw arguments against treating this as a Security Consideration. I saw opinions in favour of deprecating the construct, but I am not sure if that's an opinion for or against the health warning (since the definition of deprecation is loose here). In any case, even if you count those as votes against , I still see rough consensus. Lisa The IESG reviewed http://www.ietf.org/internet-drafts/draft- crocker-rfc4234bis-00.txt for publication as Internet Standard and would like to know if there is consensus to recommend against the use of LWSP in future specifications, as it has caused problems recently in DKIM and could cause problems in other places. Some discussion on this point already: - http://www1.ietf.org/mail-archive/web/ietf/current/msg46048.html - http://www1.ietf.org/mail-archive/web/discuss/current/msg00463.html - http://mipassoc.org/pipermail/ietf-dkim/2007q1/007295.html - https://datatracker.ietf.org/public/pidtracker.cgi? command=view_commentid=66440 (in this tracker comment, Chris Newman recommended to remove LWSP, but for backward-compatibility it's probably better to keep it and recommend against use) Thanks for your input, Lisa Dusseault ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: [ietf-dkim] Re: Use of LWSP in ABNF -- consensus call
On Thu, 17 May 2007, John C Klensin wrote: After all CRLF Thing SPACECRLF could case similar problems if some construction permitted it ... This is not news. There have for a long time been problems with significant trailing space, which is why CRLF 1*WSP CRLF in a header is part of the obs- syntax of 2822, and why quoted-printable encodes WSP at the end of a line. ... and defining a grammar that would prohibit any SPACECRLF construction isn't easy in ABNF for reasons that have nothing to do with LWSP. This is simply incorrect. It's trivial to define a whitespace construction that only allows CRLF at the beginning of a sequence: NTWSP = [CRLF] 1*WSP ; non-trailing white space Tony. -- f.a.n.finch [EMAIL PROTECTED] http://dotat.at/ SHANNON: SOUTHWEST VEERING WEST 7 TO SEVERE GALE 9, PERHAPS STORM 10 LATER. VERY ROUGH OR HIGH. RAIN OR SQUALLY SHOWERS. MODERATE OR GOOD. ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: [ietf-dkim] Re: Use of LWSP in ABNF -- consensus call
--On Friday, 18 May, 2007 09:00 +0100 Tony Finch [EMAIL PROTECTED] wrote: On Thu, 17 May 2007, John C Klensin wrote: After all CRLF Thing SPACECRLF could case similar problems if some construction permitted it ... This is not news. There have for a long time been problems with significant trailing space, which is why CRLF 1*WSP CRLF in a header is part of the obs- syntax of 2822, and why quoted-printable encodes WSP at the end of a line. ... and defining a grammar that would prohibit any SPACECRLF construction isn't easy in ABNF for reasons that have nothing to do with LWSP. This is simply incorrect. It's trivial to define a whitespace construction that only allows CRLF at the beginning of a sequence: NTWSP = [CRLF] 1*WSP ; non-trailing white space Sure. Except that much, if not most, of our textual descriptions of these protocols describes lines, and line-like, constructions as _ending_ in CRLF. Moving to starting in CRLF creates a conceptual difference between prose definition and formal syntax, which strikes me as a bad idea. Of course, in the above, since [CRLF] is optional, 1*WSP alone satisfies the production as written and still does not prevent CRLF space space space CRLF unless _all_ productions are written CRLF first... or one relies on the comment as a restriction. And a comment as the restriction --in the prose-- is exactly what has been suggested, IMO. john ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: [ietf-dkim] Re: Use of LWSP in ABNF -- consensus call
On Fri, 18 May 2007, John C Klensin wrote: On Friday, 18 May, 2007 09:00 +0100 Tony Finch [EMAIL PROTECTED] wrote: NTWSP = [CRLF] 1*WSP ; non-trailing white space Sure. Except that much, if not most, of our textual descriptions of these protocols describes lines, and line-like, constructions as _ending_ in CRLF. Moving to starting in CRLF creates a conceptual difference between prose definition and formal syntax, which strikes me as a bad idea. Don't do that then. The suggestion I made above was for a no-trailing- whitespace variant of 2822's FWS, and as such it would be used in a grammar inside a logical line, not at either end. You seem to be inventing problems that might occur in badly-designed grammars, and as such these are bugs in the grammars and not due to limitations of ABNF (which, after all, is equivalent to any other notation for describing context-free grammars). Of course, in the above, since [CRLF] is optional, 1*WSP alone satisfies the production as written and still does not prevent CRLF space space space CRLF Um, no, because WSP = SP / HTAB Tony. -- f.a.n.finch [EMAIL PROTECTED] http://dotat.at/ SOUTHEAST ICELAND: CYCLONIC, BECOMING NORTHERLY GALE 8 OR SEVERE GALE 9, DECREASING 6 OR 7. ROUGH OR VERY ROUGH, OCCASIONALLY HIGH. SHOWERS. MODERATE OR GOOD. ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: [ietf-dkim] Re: Use of LWSP in ABNF -- consensus call
Keith == Keith Moore [EMAIL PROTECTED] writes: Keith it could be argued that the best thing to do is to remove Keith ALL of the rules from the ABNF spec, leaving only the Keith language definition and examples. (actually I think I did Keith argue this sometime around 1996, but I'm too lazy to search Keith through old email to find it. I think this would be too big of a change going from draft to full given our experience. If we had huge tracts of problems with the rules, it might be a different situation. ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: [ietf-dkim] Re: Use of LWSP in ABNF -- consensus call
--On Tuesday, 15 May, 2007 11:27 -0700 Dave Crocker [EMAIL PROTECTED] wrote: Were we to deprecate every feature in IETF specifications that get mis-implemented a couple of times over 10 years, I suspect much of our technology would be deprecated... IMO, and at the risk of again agreeing with Dave, this is the issue for me. If we have inconsistent uses of terminology across documents that are supposed to be using the same, standardized, term, then that is a problem with our review process. If the term is explicitly standardized in one of the documents, that is the definition; things that use the term incorrectly should be candidates for fixing. By contrast, if we consider misused sometimes or ambiguous with other uses as a sufficient condition for deprecating the term itself, then we have a long list of terms to deprecate in front of us, almost certainly starting with IP, which refers to several different protocols, a protocol layer, and something that often involves lawyers. I think some warning language about safe and unsafe contexts may be in order for this construction but I expect that everyone who is arguing for deprecating it entirely (or inserting a strong don't use this statement) will be making a case for similar language the next time an IPv6 document comes up for review. john ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: Design of metalanguages (was: Re: Use of LWSP in ABNF -- consensus call)
On Wed, 16 May 2007, John C Klensin wrote: I find grammatical metalanguages more understandable (and readable) when there is exactly one way to write a given construction (for example, no implied terms in ranges), when the constructions are very precisely defined, and when the number of constructions is kept to a clear minimum. However you then go on to say that you like W-grammars, which tend to be highly arcane, and ISO 14977, which includes lots of alternate puctuation symbols. In fact, ISO 14977 has other disadvantages compared to RFC 4234: it has a more cluttered syntax and weaker repetition operators. It also has an exception operator which is extremely tricky: it has to be limited in a clever language-theoretical way that is necessary to keep grammars context-free, and it's very difficult to implement in a parser (almost no parser-generators have a similar operator). By contrast, all the operators of RFC 4234 are easy to implement in a parser or to desugar into what your favourite parser generator supports. It is worth noting that ABNF's line-oriented structure makes ABNF much harder to construct in xml2rfc than is really reasonable. That's a strange thing to say. The only place ABNF has strong opinions about whitespace is at the end of a rule or a comment, where it requires a newline that any sensible person would include anyway. Viewed that way, ABNF, at least as the IETF tries to use it, tends to encourage those in-between definitions: harder to understand than an informal description that relies a little bit less on syntax and more on prose, but not nearly as precise as a formal semantic definition that largely eliminates the need for the prose. I tend to agree that the semi-formal use of ABNF is unhelpful - e.g. the treatment of LWS in HTTP, where some grammar productions have implied *LWS between tokens and some do not, making the grammar useless as a complete formal description of the syntax. However this does not imply that ABNF is broken and should not be on the standards track, and switching to a different formal syntax would not fix this problem. (Especially one like ISO 14977 which is essentially equivalent to ABNF.) The problem that started this thread is that the LWSP rule inherited from the old message header format is too liberal. This is a problem for protocols that choose to use this syntax, independenly of whether they describe the syntax in ABNF. It would help future users of ABNF if the specification did not implicitly endorse syntax that we now know to be unwise. Tony. -- f.a.n.finch [EMAIL PROTECTED] http://dotat.at/ TYNE DOGGER: VARIABLE 3 OR 4 BECOMING SOUTHWEST 5 OR 6. SLIGHT OR MODERATE. OCCASIONAL RAIN. MODERATE OR GOOD. ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: [ietf-dkim] Re: Use of LWSP in ABNF -- consensus call
John == John C Klensin [EMAIL PROTECTED] writes: John --On Tuesday, 15 May, 2007 11:27 -0700 Dave Crocker John [EMAIL PROTECTED] wrote: Were we to deprecate every feature in IETF specifications that get mis-implemented a couple of times over 10 years, I suspect much of our technology would be deprecated... John IMO, and at the risk of again agreeing with Dave, this is John the issue for me. John If we have inconsistent uses of terminology across documents John that are supposed to be using the same, standardized, term, John then that is a problem with our review process. If the term John is explicitly standardized in one of the documents, that is John the definition; things that use the term incorrectly should John be candidates for fixing. I don't see why the standardized definition is the obvious right place to fix things. I thought we were committed to running code. To me, one implication of that commitment is that sometimes the right fix is for the spec to change rather than the implementations. In a terminology conflict, this often involves moving away from the term that has two conflicting uses to terms that are more clear. Ultimately cases like this should be evaluated based on whether the final result is more clear overall. ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: [ietf-dkim] Re: Use of LWSP in ABNF -- consensus call
Sam, Ultimately cases like this should be evaluated based on whether the final result is more clear overall. What about protecting the installed base for the existing spec? In other words, your based on contains a single criterion, for an environment that typically requires multiple. And the current situation is certainly one of those. Changing the specification makes sense when it does not yet have much momentum, or when the problem is basic and severe. For technology that has been in widespread use for 10-30 years, and an extremely small number of problem reports, changing the specification looks like exactly the wrong decision. -- Dave Crocker Brandenburg InternetWorking bbiw.net ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: [ietf-dkim] Re: Use of LWSP in ABNF -- consensus call
--On Thursday, 17 May, 2007 12:42 -0400 Sam Hartman [EMAIL PROTECTED] wrote: I don't see why the standardized definition is the obvious right place to fix things. I thought we were committed to running code. To me, one implication of that commitment is that sometimes the right fix is for the spec to change rather than the implementations. In a terminology conflict, this often involves moving away from the term that has two conflicting uses to terms that are more clear. Ultimately cases like this should be evaluated based on whether the final result is more clear overall. Sam, Two observations; I hope you don't think they are contradictory. (1) We regularly get ourselves into intellectual and procedural difficulties by treating specifications about how protocol specifications are written as if they were protocol specifications. When we try to avoid that, we get ourselves into worse problems. Using rules that are more or less arbitrary, we make some of these documents Proposed Standards and then try to progress them, we make others into BCPs and, now, we make still others into IONs. If we are going to standardize a definitional requirement or method -- whether it is ABNF or IPR boilerplate or something -- we need to get it right as a self-contained definition and then live with it. We should certainly revise and replace it if it turns out to be unworkable (as has happened with IPR work) or if the definition turns out to be inadequate to permit an unambiguous interpretation (that issue spills over into my second observation, below). But, once other specifications start to depend on the definitions that are there, and show those definitions to be adequate, we should not be talking about deprecating definitions unless we are prepared to that was wrong, we need to start over (even though some of the older material may still be useful).Again, please note the similarity to the IPR work. (2) If we pretend that the ABNF metalanguage and definitions are actually a protocol specification, then we need to evaluate it as one. Then, we have the following criteria (which we usually don't state quite this precisely): (i) Is the definition good enough that interoperable implementations are possible? (ii) Do people care enough about the construct to actually use it in ways that show it is useful? Now, neither of those rules prevents non-conforming implementations. We may notice that those exist, but our concern is only about implementations that appear to conform to the spec and are (or are not) interoperable. If non-conforming implementations happen by accident because the text isn't clear enough, we try to clarify the text. But we don't say well, there are non-conforming implementations, so the spec is broken. That would make no sense at all, at least to me. The answer to (i) appears to be yes. There are lots of conforming cases. And the answer to (ii) is, as Dave as pointed out repeatedly, about 30 years worth. Is this construction dangerous if used in inappropriate contexts? Sure. Does that justify a warning note to the unwary? Probably. Is it possible to implement other things and call them by the same name (i.e., create a non-conforming implementation)? Of course. Should that invalidate the definition? Not if we want to have anything left if the principle were applied broadly. john ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: Use of LWSP in ABNF -- consensus call
John C Klensin wrote: I would have no problems adding a comment to any construction (including built-in productions) in ABNF that has proven dangerous warning people that they should understand it and its consequences before they use it. That's the intention of the proposed security considerations. It could be placed elsewhere, or reduced to there be dragons if that's appropriate language for an STD (I'd like it), these proposals have the same effect of deprecating LWSP from my POV. I see those options as very different from deprecating something that is used successfully and correctly in a number of standards and incorporated into them by reference. Then our terminology of deprecating is different, but neither there be dragons nor the longer security blurb used the verb deprecate explicitly. Since it is in use Do you know a single case of a x234-LWSP intentionally allowing apparently empty lines ? I'd guess that authors just copied it without considering this effect, or because their protocol doesn't need any really empty lines with a different meaning. While I'm at it, the 4234 definition of LWSP doesn't use the grouping notation recommended in 4234: *(WSP / CRLF WSP) is the same as the qstrongly advised/q *(WSP / (CRLF WSP)) Frank ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: [ietf-dkim] Re: Use of LWSP in ABNF -- consensus call
On Thu, 17 May 2007, John C Klensin wrote: Is this construction dangerous if used in inappropriate contexts? Sure. Does that justify a warning note to the unwary? Probably. Is it possible to implement other things and call them by the same name (i.e., create a non-conforming implementation)? Of course. Should that invalidate the definition? Not if we want to have anything left if the principle were applied broadly. +1 Tony. -- f.a.n.finch [EMAIL PROTECTED] http://dotat.at/ ROCKALL: SOUTHWEST 6 TO GALE 8, INCREASING SEVERE GALE 9, PERHAPS STORM 10 LATER. VERY ROUGH OR HIGH. SHOWERS. GOOD. ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: Design of metalanguages (was: Re: Use of LWSP in ABNF -- consensus call)
On May 17, 2007, at 9:29 AM, Tony Finch wrote: It would help future users of ABNF if the specification did not implicitly endorse syntax that we now know to be unwise. +1 Especially when not germane to ABNF definitions. The construct should stand on its own when used. Perhaps labeled as ThereBeDragons. :) -Doug ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: [ietf-dkim] Re: Use of LWSP in ABNF -- consensus call
John == John C Klensin [EMAIL PROTECTED] writes: John If we are going to standardize a definitional requirement or John method -- whether it is ABNF or IPR boilerplate or something John -- we need to get it right as a self-contained definition John and then live with it. We should certainly revise and John replace it if it turns out to be unworkable (as has happened John with IPR work) or if the definition turns out to be John inadequate to permit an unambiguous interpretation (that John issue spills over into my second observation, below). But, John once other specifications start to depend on the definitions John that are there, and show those definitions to be adequate, John we should not be talking about deprecating definitions John unless we are prepared to that was wrong, we need to start John over (even though some of the older material may still be John useful). Again, please note the similarity to the IPR John work. Right. Here, I don't think the definition is wrong, I just think the term being defined is wrong. We proposed a definition for a useful concept. The word we chose (LWSP) stuck in some places but not in others. IN fact other people used the same word for a different although related concept. sufficiently so that the definition we proposed in ABNF is not the most common definition in our standards. Clearly we should not invalidate existing uses of that term. Clearly we do need a definition for the term: it is being used usefully. I think that in this instance, the value of future clarity justifies coming up with a new term that will unambiguously mean what LWSP means in ABNF today. That term will have to start at proposed standard. LWSP will need to continue in ABNF. I see a desire to document our operational experience with the word: many people took this word and used it to mean something else. Perhaps to avoid confusion you should consider whether your use of the word is a good idea. I think there is significant harm in choosing not to document this operational experience when advancing standards. After all, we both agree that it is this experience with running code that gives the IETF value. ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: [ietf-dkim] Re: Use of LWSP in ABNF -- consensus call
Sam, with one small exception, I think we are in complete agreement. The exception is noted below... --On Thursday, 17 May, 2007 15:32 -0400 Sam Hartman [EMAIL PROTECTED] wrote: Right. Here, I don't think the definition is wrong, I just think the term being defined is wrong. We proposed a definition for a useful concept. The word we chose (LWSP) stuck in some places but not in others. IN fact other people used the same word for a different although related concept. sufficiently so that the definition we proposed in ABNF is not the most common definition in our standards. Counting uses in this way so as to determine what is or is not the most common definition is as tricky in our environment as counting votes. For the same reason, I'd rather that we avoid it when we can. However, if we do need to count, I think we have to use some sort of system that attributes more weight to documents that use the terminology that have been, effectively, at full standard before some of the specifications that use different definitions were even dreamt of. I don't know how to assign those weights, which, recursively, is why I don't like counting. But I don't think you can count up a handful of recent misuses and then make an inference that the base definition needs to be retired (which you haven't done, but several others have), or even that it is not appropriately common. I think that in this instance, the value of future clarity justifies coming up with a new term that will unambiguously mean what LWSP means in ABNF today. That term will have to start at proposed standard. Pragmatically, perhaps yes. But what I keep coming back to is that it appears to me to be perfectly clear and unambiguous what LWSP means in ABNF today. That is what is written into the spec. As far as I can tell, we are having a discussion about two other things: (1) Other specifications that use the term LWSP to refer to something different from what is unambiguously defined in the ABNF spec. (2) Places where people might be tempted to use LWSP but where they have discovered that LWSP is either inappropriate or risky. The second group clearly needs new and appropriate terminology for what they do need and use. The first group is, IMO, just broken. LWSP will need to continue in ABNF. I see a desire to document our operational experience with the word: many people took this word and used it to mean something else. Perhaps to avoid confusion you should consider whether your use of the word is a good idea. I think there is significant harm in choosing not to document this operational experience when advancing standards. After all, we both agree that it is this experience with running code that gives the IETF value. Again, I have no problems with making carefully-considered comments about use and misuse. I think it is a fine idea as long as those comments do not imply that the misuses are a failure in the base specification. And I think everyone should consider a stopping rule, lest, as I indicated in my deliberately extreme example, we discover that we want to retire, replace, or update IP because of some use in the legal community that precedes the use in network protocols by many years. john ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: [ietf-dkim] Re: Use of LWSP in ABNF -- consensus call
--On 17. mai 2007 15:32 -0400 Sam Hartman [EMAIL PROTECTED] wrote: Right. Here, I don't think the definition is wrong, I just think the term being defined is wrong. We proposed a definition for a useful concept. Actually we defined a concept (LWSP) in a way that turned out to be much more troublesome in several contexts than we thought it would be (because it allowed two different versions of blank lines to have different semantics). The word we chose (LWSP) stuck in some places but not in others. IN fact other people used the same word for a different although related concept. sufficiently so that the definition we proposed in ABNF is not the most common definition in our standards. Clearly we should not invalidate existing uses of that term. Clearly we do need a definition for the term: it is being used usefully. I don't agree with the meaning I get from this statement. The problem is that the construct that ABNF calls LWSP causes problems in protocols that use it. This problem is independent of the name of the construct; the problem is in defining a grammar where the sequence CRLFCRLF has a different meaning than CRLFSPACECRLF. Some protocols have addressed this problem by defining their own construct, and have used the term LWSP to refer to it - something that is legal by ABNF rules, but can cause people who read multiple specs to become confused. There seems to be some reason to think that it's useful to warn people that there's reasons not to use the construct that is defined as LWSP in the ABNF spec. The revised version of the ABNF spec is one possible place to put that warning. I think that in this instance, the value of future clarity justifies coming up with a new term that will unambiguously mean what LWSP means in ABNF today. That term will have to start at proposed standard. LWSP will need to continue in ABNF. I agree with the last statement. I see a desire to document our operational experience with the word: many people took this word and used it to mean something else. Perhaps to avoid confusion you should consider whether your use of the word is a good idea. I think there is significant harm in choosing not to document this operational experience when advancing standards. After all, we both agree that it is this experience with running code that gives the IETF value. Fully agree with this statement too, but since I don't agree with your interpretation (as given above) of what the problem is, I can't be sure that my agreement with this statement means anything Harald ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: [ietf-dkim] Re: Use of LWSP in ABNF -- consensus call
Harald, I'm happy to accept your interpretation of the problem. However it also leads me to the conclusion that documenting possible reasons not to use ABNF's LWSP concept, or documenting implications of that rule would be a good idea. I also believe that documenting experience with a spec in future versions of that spec as it advances is reasonable. ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: [ietf-dkim] Re: Use of LWSP in ABNF -- consensus call
Dave == Dave Crocker [EMAIL PROTECTED] writes: Dave Sam, Ultimately cases like this should be evaluated based on whether the final result is more clear overall. Dave What about protecting the installed base for the existing Dave spec? I think that is not a useful criteria when we are talking about an informative note. I think that criteria matters somewhat more when we are talking about depricating a feature but retaining it, although even then I think the bar would be reasonably low. The installed base will continue to work. I think that criterian is very important if we were talking about removing a rule. For that reason I do not favor removing the LWSP rule. ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: [ietf-dkim] Re: Use of LWSP in ABNF -- consensus call
Sam Hartman wrote: Ultimately cases like this should be evaluated based on whether the final result is more clear overall. Dave What about protecting the installed base for the existing Dave spec? I think that is not a useful criteria when we are talking about an informative note. I think that criteria matters somewhat more when we are talking about depricating a feature but retaining it, although even then I think the bar would be reasonably low. The installed base will continue to work. I think you are assuming a more constrained discussion than what I've been seeing on this thread. The thread has discussed everything from removing the rule, to redefining it, to declaring it deprecated, to adding some commentary text. It appears you are only talking about the last, although I for one missed that. (For reference, when I said change the specification I mean normative change.) Although I've seen some postings against even having a comment added to the text, my own reading of the postings is that there is a reasonable consensus that it would be ok. My own view is that comments can be helpful and are, at worst, typically rather benign. Indeed, the IETF approach towards specification writing is rather friendly towards including whatever comments folk feel might be helpful, modulo the obvious danger that too many comments can wind up obscuring a document. (And, no, I do not think that that is a danger here.) d/ -- Dave Crocker Brandenburg InternetWorking bbiw.net ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: [ietf-dkim] Re: Use of LWSP in ABNF -- consensus call
I think redefining the rule would require recycling at proposed. I think it would be confusing and harmful to do so. I think removing the rule would is allowed by the process (and would require updates in referencing specs as they advance but would not break anything). I think doing so would be harmful and is not supported by consensus. I do not object to deprecating the rule although I'm not completely convinced doing so is a great idea.I think it is clearly allowed if there is consensus. ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: [ietf-dkim] Re: Use of LWSP in ABNF -- consensus call
On Thu, 17 May 2007, John C Klensin wrote: (1) Other specifications that use the term LWSP to refer to something different from what is unambiguously defined in the ABNF spec. [This] group is, IMO, just broken. I agree with your sentiment but sadly there's a lot of old stuff that's broken by this criterion. RFC 733 defined LWSP-char to mean what RFC 2234 calls WSP, and linear-white-space to mean what is now LSWP. Fortunately these old definitions have fallen out of use. There's also HTTP and SIP which call the problematic production LWS instead of LWSP. MEGACO uses ABNF with its own terminal definitions instead of referring to ABNF appendix B. CGI uses its own ABNF variant. It would be nice if the progression of ABNF to a full standard reduces this Babel. This implies that (a) ABNF should be used to describe syntax in preference to any other BNF-alike, to avoid anomalies like CGI; (b) specifications should not define a production with the same name as one in appendix B but with a different expansion, to avoid anomalies like MEGACO; (c) specifications should not define a production with the same expansion as one in appendix B but with a different name, to avoid anomalies like HTTP and SIP; (d) any production like LWSP should be discouraged because of problems with lossage related to trailing whitespace. Tony. -- f.a.n.finch [EMAIL PROTECTED] http://dotat.at/ ROCKALL: SOUTHWEST 6 TO GALE 8, INCREASING SEVERE GALE 9, PERHAPS STORM 10 LATER. VERY ROUGH OR HIGH. SHOWERS. GOOD. ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: [ietf-dkim] Re: Use of LWSP in ABNF -- consensus call
On May 17, 2007, at 2:27 PM, Dave Crocker wrote: I think you are assuming a more constrained discussion than what I've been seeing on this thread. The thread has discussed everything from removing the rule, to redefining it, to declaring it deprecated, to adding some commentary text. I think that only the later was suggested. Declaring the macro deprecated would not remove the definition or redefine it. Any additional comments should simply indicate this construct should be used with caution and defined locally with a different mnemonic. On the bright side, this does not impact the ABNF definition. -Doug ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: [ietf-dkim] Re: Use of LWSP in ABNF -- consensus call
--On Thursday, 17 May, 2007 21:52 +0200 Harald Tveit Alvestrand [EMAIL PROTECTED] wrote: I don't agree with the meaning I get from this statement. The problem is that the construct that ABNF calls LWSP causes problems in protocols that use it. This problem is independent of the name of the construct; the problem is in defining a grammar where the sequence CRLFCRLF has a different meaning than CRLFSPACECRLF. ... Interesting. I don't think that is a problem with the grammar, and think it would be rather hard to define a grammar that would not permit that situation. After all CRLF Thing SPACECRLF could case similar problems if some construction permitted it and defining a grammar that would prohibit any SPACECRLF construction isn't easy in ABNF for reasons that have nothing to do with LWSP. Instead, I see the problem as using the grammar to define situations equivalent to LWSP [ optional-stuff ] CRLF as compared to LWSP AtLeastOneRequiredThing CRLF or [ LWSP optional-stuff ] CRLF I don't see either of the latter as problematic. john ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: [ietf-dkim] Re: Use of LWSP in ABNF -- consensus call
it could be argued that the best thing to do is to remove ALL of the rules from the ABNF spec, leaving only the language definition and examples. (actually I think I did argue this sometime around 1996, but I'm too lazy to search through old email to find it. I'm actually surprised that a problem with one of those definitions has taken this long to crop up.) the rules could then be moved to a separate document, probably with status informational. I don't personally see any problem with an informational document being used to define terms that are referenced in standards track publications - so long as the document being referenced is not subject to change.the referencing document should be evaluated as if the definitions were incorporated into that document. IMHO, sometimes we try too hard to make things fit into the standards track. application of the proposed/draft/full criteria to ABNF has always been a bit odd, because ABNF defines a notation rather than a protocol. at one time we thought about evaluating interoperability of ABNF by seeing if there were multiple implementations (i.e. parser generators) that when given the same grammar, recognized the same language. but this would be kind of meaningless given that ABNF is (in my experience) almost never used as input to a parser generator for the purpose of generating a protocol implementation. which might be quite unfortunate. I think redefining the rule would require recycling at proposed. I think it would be confusing and harmful to do so. I think removing the rule would is allowed by the process (and would require updates in referencing specs as they advance but would not break anything). I think doing so would be harmful and is not supported by consensus. I do not object to deprecating the rule although I'm not completely convinced doing so is a great idea.I think it is clearly allowed if there is consensus. ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: Use of LWSP in ABNF -- consensus call
John C Klensin wrote: LWSP AtLeastOneRequiredThing CRLF or [ LWSP optional-stuff ] CRLF I don't see either of the latter as problematic. Depends on the protocol. Your constructs match SP CRLF SP CRLF SP AtLeastOneRequiredThing CRLF SP CRLF SP CRLF SP optional-stuff CRLF . If you do this e.g. in a mail or HHTP header, where CRLF CRLF means end of header, then any tool like a text editor or other UA trying to reuse the header field as is while silently removing trailing white space would end up with CRLF CRLF SP AtLeastOneRequiredThing CRLF CRLF CRLF SP optional-stuff CRLF ..^ Frank ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: Use of LWSP in ABNF -- consensus call
--On Tuesday, 15 May, 2007 16:00 -0700 Douglas Otis [EMAIL PROTECTED] wrote: The use under RFC2530 is a bit vague (with LWSP wrapping); likewise under RFC3501 (otherwise treat SP as being equivalent to LWSP). The use under RFC4646 has caused known problems. This would seem to justify deprecation, IMHO. Agreed. From a standards standpoint, half a dozen definitions for an ABNF mnemonic is absurd. Perhaps something like: The LWSP mnemonic has been deprecated and should not be used in future drafts. Explicit definitions based upon different mnemonics should be used instead. If possible, syntax should guard against possible security concerns related to visual deceptions. Doug, John, It seems to me that we have two separate issues here (I'm not even going to go so far as problems): (1) Some documents have used the term LWSP in a way that is not strictly conformant with the definition in the ABNF document. Unless the definition is unclear, that is a problem for those documents -- and the review and approval process that permitted them to slip through with non-conforming definitions -- and not for the ABNF spec. It seems to me we fix that by changing those documents to use different production names with local definitions, not by deprecating features of ABNF. (2) What I learned back when I was doing definitional work for programming languages was that one wants one's basic syntax and syntactic metalanguages to be as simple as possible, with a minimum of macro constructions and with a minimum of defined terminals. From that point of view, it is easy to argue that ABNF has just gotten too complex, both as the result of trying to formalize some characteristics of 822 while maintaining a single-pass syntax evaluator and possibly as a second-system effect. That, however, is an argument that ABNF itself is broken and should not be on the standards track. It is not an argument for trying to fine-tune it by deprecating a production or two. The broken argument itself was made by a few of us back when RFC 2234 was being evaluated. We lost and, at this point, it would take a lot more than one sometimes-misused construction to justify tampering with it, even to the extent of selectively deprecating features. john ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: Use of LWSP in ABNF -- consensus call
Tony Hansen said: I share your concerns about removing rules that are already in use -- that would generally be a bad thing. However I'm interested in the consensus around whether a warning or a deprecation statement would be a good thing. LWSP has a valid meaning and use, and its being misapplied somewhere doesn't make that meaning and usage invalid. I could see a note being added. However, anything more than that is totally inappropriate. +1 Frank's text in http://www1.ietf.org/mail-archive/web/ietf/current/msg46048.html would be fine: Authors intending to use the LWSP (linear white space) construct should note that it allows apparently empty lines consisting only of trailing white space, semantically different from really empty lines. Some text editors and other tools are known to remove any trailing white space silently, and therefore the use of LWSP in syntax is not recommended. However, it doesn't belong in security considerations. What about moving LSWP, and this text, to a separate section of Annex B: B.3 Deprecated constructs? -- Clive D.W. Feather | Work: [EMAIL PROTECTED] | Tel:+44 20 8495 6138 Internet Expert | Home: [EMAIL PROTECTED] | Fax:+44 870 051 9937 Demon Internet | WWW: http://www.davros.org | Mobile: +44 7973 377646 THUS plc|| ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: Use of LWSP in ABNF -- consensus call
On Mon, 14 May 2007, Lisa Dusseault wrote: The IESG reviewed http://www.ietf.org/internet-drafts/draft-crocker-rfc4234bis-00.txt for publication as Internet Standard and would like to know if there is consensus to recommend against the use of LWSP in future specifications, as it has caused problems recently in DKIM and could cause problems in other places. LWSP was one of the factors in the SASL WG's abandoning of the DIGEST-MD5 revision. To be precise, the complexity inherent in the 822-style n#rule syntax (the emulation of which in modern ABNF uses LWSP) was found to be a source of implementation errors and interoperability issues. Part of the discussion can be seen at: http://www.imc.org/ietf-sasl/mail-archive/msg02752.html While I don't think removing LWSP from the ABNF RFC is good idea, adding a comment to its definition in section B.1 that warns protocol designers to consider carefully whether they really need or want LWSP instead of *WSP does seem appropriate. Calling out that LWSP permits lines containing just whitespace may help drive home the issue. Philip Guenther Sendmail, Inc ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: Use of LWSP in ABNF -- consensus call
At 5:35 PM -0700 5/14/07, Lisa Dusseault wrote: I share your concerns about removing rules that are already in use -- that would generally be a bad thing. However I'm interested in the consensus around whether a warning or a deprecation statement would be a good thing. A warning would be very useful, and a deprecation or removal would be very bad because either would cause people to go to the earlier documents to see what LWSP meant earlier. ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
RE: [ietf-dkim] Re: Use of LWSP in ABNF -- consensus call
Add a warning note Bill Oxley Messaging Engineer Cox Communications 404-847-6397 -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of [EMAIL PROTECTED] Sent: Tuesday, May 15, 2007 3:03 PM To: Tony Hansen Cc: Apps Discuss; IETF General Discussion Mailing List; [EMAIL PROTECTED] Subject: [ietf-dkim] Re: Use of LWSP in ABNF -- consensus call Lisa Dusseault wrote: I share your concerns about removing rules that are already in use -- that would generally be a bad thing. However I'm interested in the consensus around whether a warning or a deprecation statement would be a good thing. LWSP has a valid meaning and use, and its being misapplied somewhere doesn't make that meaning and usage invalid. I could see a note being added. However, anything more than that is totally inappropriate. Full agreement with Tony here. Ned ___ NOTE WELL: This list operates according to http://mipassoc.org/dkim/ietf-list-rules.html ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: Use of LWSP in ABNF -- consensus call
On Wed, 16 May 2007, John C Klensin wrote: It seems to me that we have two separate issues here (I'm not even going to go so far as problems): (1) Some documents have used the term LWSP in a way that is not strictly conformant with the definition in the ABNF document. (2) From that point of view, it is easy to argue that ABNF has just gotten too complex, both as the result of trying to formalize some characteristics of 822 while maintaining a single-pass syntax evaluator and possibly as a second-system effect. I thought the problem is that protocols that have used LWSP correctly have had too many interop problems, so they have replaced it with a simpler rule such as FWS. I'm surprised you say ABNF has become too complex. It's hardly changed apart from removal of the # rule, and if you took anything else out it would lead to rather less readable grammars. Tony. -- f.a.n.finch [EMAIL PROTECTED] http://dotat.at/ NORTH FITZROY SOLE LUNDY FASTNET: WEST OR NORTHWEST 4 OR 5, OCCASIONALLY 6 OR 7. MODERATE, OCCASIONALLY ROUGH. OCCASIONAL RAIN OR DRIZZLE. GOOD BECOMING MODERATE OR POOR. ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: Use of LWSP in ABNF -- consensus call
I suggest that a version of this sentence goes into 4234bis: I thought the problem is that protocols that have used LWSP correctly have had too many interop problems, so they have replaced it with a simpler rule such as FWS. ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: Use of LWSP in ABNF -- consensus call
On May 15, 2007, at 1:10 AM, Clive D.W. Feather wrote: Tony Hansen said: I share your concerns about removing rules that are already in use -- that would generally be a bad thing. However I'm interested in the consensus around whether a warning or a deprecation statement would be a good thing. LWSP has a valid meaning and use, and its being misapplied somewhere doesn't make that meaning and usage invalid. I could see a note being added. However, anything more than that is totally inappropriate. +1 Frank's text in http://www1.ietf.org/mail-archive/web/ietf/current/msg46048.html would be fine: Authors intending to use the LWSP (linear white space) construct should note that it allows apparently empty lines consisting only of trailing white space, semantically different from really empty lines. Some text editors and other tools are known to remove any trailing white space silently, and therefore the use of LWSP in syntax is not recommended. However, it doesn't belong in security considerations. Discarding of lines is likely in response to some type of exploit. The consideration for not using LWSP would be in regard with how security requirements may create incompatibilities. This is the correct section. What about moving LSWP, and this text, to a separate section of Annex B: B.3 Deprecated constructs? Agreed. That would also be appropriate. Another problem regarding LWSP is in regard to _many_ differing definitions. A profusion of differing definitions alone becomes a valid reason to deprecate the mnemonic. This definition represents a poor practice as related to security which should not be facilitated through standardization. By removing this problematic construct, better solutions are more likely to be found. At least (ab)use of the mnemonic will have been discouraged. Any continued use of this mnemonic should be discouraged and the note added should clarify one of the reasons for this mnemonic being deprecated is specifically due to its varied and checkered meanings in other drafts. -Doug ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: Use of LWSP in ABNF -- consensus call
On May 16, 2007, at 5:19 AM, John C Klensin wrote: Doug, John, It seems to me that we have two separate issues here (I'm not even going to go so far as problems): (1) Some documents have used the term LWSP in a way that is not strictly conformant with the definition in the ABNF document. Unless the definition is unclear, that is a problem for those documents -- and the review and approval process that permitted them to slip through with non-conforming definitions -- and not for the ABNF spec. It seems to me we fix that by changing those documents to use different production names with local definitions, not by deprecating features of ABNF. This is not deprecating a feature of ABNF other than deprecating a specific, confusing, and ill-advised mnemonic found in a pre-defined macro library. As this construct is often ill-advised, there is little justification for it to be included in the ABNF pre-defined macro library. This change does not preclude any construct, it only affects ease of use. This should invoke greater care and forethought. (2) What I learned back when I was doing definitional work for programming languages was that one wants one's basic syntax and syntactic metalanguages to be as simple as possible, with a minimum of macro constructions and with a minimum of defined terminals. From that point of view, it is easy to argue that ABNF has just gotten too complex, both as the result of trying to formalize some characteristics of 822 while maintaining a single-pass syntax evaluator and possibly as a second-system effect. While it is a poor craftsman who blames his tools, it is also difficult to justify standardizing a macro construct proven often to be problematic. When an author is looking for a macro, this construct and mnemonic should not be within a pre-defined macro library. Exclusion of LWSP does not represent any hardship. That, however, is an argument that ABNF itself is broken and should not be on the standards track. It is not an argument for trying to fine-tune it by deprecating a production or two. The broken argument itself was made by a few of us back when RFC 2234 was being evaluated. We lost and, at this point, it would take a lot more than one sometimes-misused construction to justify tampering with it, even to the extent of selectively deprecating features. Although ABNF limitations might lead to sub-optimal syntax, how does it prevent optimal syntax from being defined? This change does not tamper with the ABNF language. It only deprecates a pre-defined macro proven problematic and encumbered with differing definitions. Any future draft will not find the lack of this macro definition to represent any type of hardship. Those reading IETF drafts however will find whatever replaces this macro will be specifically defined where security concerns are more likely to have been given greater consideration. Whatever macro then commonly replaces LWSP could be given a different mnemonic and added to a subsequent version of this draft. -Doug ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Design of metalanguages (was: Re: Use of LWSP in ABNF -- consensus call)
--On Wednesday, 16 May, 2007 13:58 +0100 Tony Finch [EMAIL PROTECTED] wrote: ... I'm surprised you say ABNF has become too complex. It's hardly changed apart from removal of the # rule, and if you took anything else out it would lead to rather less readable grammars. Tony, First, my apologies in advance to everyone who doesn't care about this subject (the change of subject line is a deliberate clue). From my point of view, the IETF made its decision around ten years ago and it is questionable whether the decision then could sensibly have departed significantly from the definition is RFC 822. While I didn't like the decision then and don't like it now (perhaps the comments below will explain that), I think it would be a serious mistake to open the question up again. This is a matter of taste, and perhaps generational. I find grammatical metalanguages more understandable (and readable) when there is exactly one way to write a given construction (for example, no implied terms in ranges), when the constructions are very precisely defined, and when the number of constructions is kept to a clear minimum. All things being equal, I like the ability to construct a two-stage grammar (W-grammar) when that seems to be justified by circumstances: ABNF does not permit that, and the omission is one reason we keep having little messes about comments in protocols whose syntax is defined in ABNF. Similarly, I prefer grammars that can be written as a data stream without depending on line-ending (and indented continuation) as important indicators. It is worth noting that ABNF's line-oriented structure makes ABNF much harder to construct in xml2rfc than is really reasonable. I've also got a second problem, which is that I'm more comfortable with either informal descriptions that use formal grammars for clarification or explanation for with fully-formal grammatical descriptions (including semantics) than I am with the in-between points. Viewed that way, ABNF, at least as the IETF tries to use it, tends to encourage those in-between definitions: harder to understand than an informal description that relies a little bit less on syntax and more on prose, but not nearly as precise as a formal semantic definition that largely eliminates the need for the prose. This is definitely a personal idiosyncrasy: YMMD and probably does. For those of you who are interested in what a syntax-only metalanguage would look like that would better meet my personal criteria, I'd encourage taking a look at ISO/IEC ISO/IEC 14977:1996 Information technology -- Syntactic metalanguage -- Extended BNF. It is a bit earlier than, but roughly contemporary with, RFC 2234 and similar in enough ways that one of them should probably be referencing the other (if one can guess from timing, the ISO/BSI document should be acknowledging RFC 822). Unlike the ISO (and ISO/IEC JTC1) norms, this one is actually freely available, see the link from http://isotc.iso.org/livelink/livelink/fetch/2000/2489/Ittf_Home/ITTF.htm For comparison purposes, while the introductory material is interesting and helpful, the specification itself is only ten pages long (although one needs to adjust for smaller type and more page layout), including examples, and defines the metalanguage three different ways. But, again, this is all a matter of theorizing: I think it would be a terrible idea to try to redesign or replace ABNF at this point. john ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: Use of LWSP in ABNF -- consensus call
In response to off-line comments, Although LWSP has been placed within core rules, LWSP is _not_ a rule core to the ABNF definition of ABNF. LWSP is _not_ essential. Deprecating this macro does _not_ impact the definition of ABNF. This macro can be deprecated to ensure it will not promote use of this construct, nor should this macro be used to supplant other definitions. The LWSP jersey can be retired without damaging the definition of ABNF or otherwise limiting the future use of ABNF. -Doug ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: Use of LWSP in ABNF -- consensus call
--On Wednesday, 16 May, 2007 17:21 -0700 Douglas Otis [EMAIL PROTECTED] wrote: In response to off-line comments, Although LWSP has been placed within core rules, LWSP is _not_ a rule core to the ABNF definition of ABNF. LWSP is _not_ essential. Deprecating this macro does _not_ impact the definition of ABNF. This macro can be deprecated to ensure it will not promote use of this construct, nor should this macro be used to supplant other definitions. The LWSP jersey can be retired without damaging the definition of ABNF or otherwise limiting the future use of ABNF. Doug, if people want to do it, I would have no problems adding a comment to any construction (including built-in productions) in ABNF that has proven dangerous warning people that they should understand it and its consequences before they use it. I would have no problems if that note made it clear that use of LWSP in a context in which it could end up on a line by itself (in a context in which lines are significant) can be particularly problematic. I see those options as very different from deprecating something that is used successfully and correctly in a number of standards and incorporated into them by reference. Since it is in use, and the definition is actually quite clear, deprecating it seems completely inappropriate. john ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: Use of LWSP in ABNF -- consensus call
On May 16, 2007, at 5:47 PM, John C Klensin wrote: I would have no problems if that note made it clear that use of LWSP in a context in which it could end up on a line by itself (in a context in which lines are significant) can be particularly problematic. I see those options as very different from deprecating something that is used successfully and correctly in a number of standards and incorporated into them by reference. Since it is in use, and the definition is actually quite clear, deprecating it seems completely inappropriate. There is no single definition for LWSP across many documents. A statement that both warns about possible adverse effects and that a different label should be considered for new drafts solves both the problem of conflicting definitions AND inappropriate promotion of this construct. A statement that LWSP is deprecated does not imply it is now obsolete. Other documents which have referenced this document will simply find the definition deprecated. New drafts should consider either creating a more appropriate definition, or use a different mnemonic with their desired definition. -Doug ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: Use of LWSP in ABNF -- consensus call
On Mon, 14 May 2007, Dave Crocker wrote: Could cause problems in other places... The DKIM hiccup was the first one I'd heard about. By contrast, linear-white-space was defined in RFC733, in 1977, with RFC822 retaining that definition. It is defined in those places as essentially the same as LWSP in the current ABNF Draft Standard specification. The LWSP defined in ABNF is more like the one in HTTP than the message format one, in that 4234 allows space-only lines (it allows multiple CRLFs in LWSP) whereas 2822 does not (at most one CRLF in FWS). There is some documentation of the interoperability problems arising from the implied-LWS rule in HTTP here: http://www.and.org/texts/server-http#implicit-lws Tony. -- f.a.n.finch [EMAIL PROTECTED] http://dotat.at/ GERMAN BIGHT HUMBER THAMES: NORTHWEST BACKING SOUTHWEST 4 OR 5, OCCASIONALLY 6 OR 7 AT FIRST. MODERATE, OCCASIONALLY ROUGH IN GERMAN BIGHT. RAIN OR SHOWERS. MODERATE OR GOOD, OCCASIONALLY POOR. ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: Use of LWSP in ABNF -- consensus call
In message [EMAIL PROTECTED], Tony Hansen [EMAIL PROTECTED] writes Lisa Dusseault wrote: I share your concerns about removing rules that are already in use -- that would generally be a bad thing. However I'm interested in the consensus around whether a warning or a deprecation statement would be a good thing. LWSP has a valid meaning and use, and its being misapplied somewhere doesn't make that meaning and usage invalid. Agreed - well put. I could see a note being added. However, anything more than that is totally inappropriate. I would vote against even adding a note. It seems disproportionate to change a 10 year specification at this late stage on the basis of a single case of a misapplied, but valid, rule in another specification. Regards -- Paul Overell Internet Platform Development Manager, Thus plc ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: Use of LWSP in ABNF -- consensus call
I had followed up to Tony's note privately with Tony/Lisa/Dave yesterday, but perhaps I should have posted it here. No time like the present. I agree that technical changes to a specification as it moves from Draft to Full does not seem helpful. Although we have darned little experience actually DOING this, RFC 2026 says 4.1.3 Internet Standard A specification for which significant implementation and successful operational experience has been obtained may be elevated to the Internet Standard level. An Internet Standard (which may simply be referred to as a Standard) is characterized by a high degree of technical maturity and by a generally held belief that the specified protocol or service provides significant benefit to the Internet community. So, it seems to me that the bar for changing technical aspects of the specification needs to be very high, since this specification has been significantly implemented and successfully operated for a while now. Specifically, a deprecation statement would make sense if LWSP is ALWAYS a bad idea, but we don't seem to be saying that. So I would speak against deprecating. I believe that the IETF has a responsibility to implementers. Anytime we know there is a dragon, we should say there be dragons. So I would speak in favor of adding a warning note that clearly points to a dragon when this specification is used in a new specification. Thanks, Spencer p.s. I have to smile a little that we're talking about ABNF being used in new protocol specifications after 10-30 years, depending on who is counting. I wish ALL of our draft standards, and full standards, were as helpful to the Internet community. From: Paul Overell [EMAIL PROTECTED] In message [EMAIL PROTECTED], Tony Hansen [EMAIL PROTECTED] writes Lisa Dusseault wrote: I share your concerns about removing rules that are already in use -- that would generally be a bad thing. However I'm interested in the consensus around whether a warning or a deprecation statement would be a good thing. LWSP has a valid meaning and use, and its being misapplied somewhere doesn't make that meaning and usage invalid. Agreed - well put. I could see a note being added. However, anything more than that is totally inappropriate. I would vote against even adding a note. It seems disproportionate to change a 10 year specification at this late stage on the basis of a single case of a misapplied, but valid, rule in another specification. ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: Use of LWSP in ABNF -- consensus call
Lisa Dusseault wrote: The IESG reviewed http://www.ietf.org/internet-drafts/draft-crocker- rfc4234bis-00.txt for publication as Internet Standard and would like to know if there is consensus to recommend against the use of LWSP in future specifications, as it has caused problems recently in DKIM and could cause problems in other places. Some discussion on this point already: - http://www1.ietf.org/mail-archive/web/ietf/current/msg46048.html - http://www1.ietf.org/mail-archive/web/discuss/current/msg00463.html - http://mipassoc.org/pipermail/ietf-dkim/2007q1/007295.html - https://datatracker.ietf.org/public/pidtracker.cgi? command=view_commentid=66440 (in this tracker comment, Chris Newman recommended to remove LWSP, but for backward-compatibility it's probably better to keep it and recommend against use) I think it would be better to keep LWSP and recommending against its use. Running grep on various RFCs shows that this production is referenced in several RFCs, even in some recent ones. I don't object to Chris' idea to add FWS to the ABNF document. ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: Use of LWSP in ABNF -- consensus call
Paul Overell [EMAIL PROTECTED] wrote: In message [EMAIL PROTECTED], Tony Hansen [EMAIL PROTECTED] writes Lisa Dusseault wrote: I share your concerns about removing rules that are already in use -- that would generally be a bad thing. However I'm interested in the consensus around whether a warning or a deprecation statement would be a good thing. LWSP has a valid meaning and use, and its being misapplied somewhere doesn't make that meaning and usage invalid. Agreed - well put. I could see a note being added. However, anything more than that is totally inappropriate. I would vote against even adding a note. It seems disproportionate to change a 10 year specification at this late stage on the basis of a single case of a misapplied, but valid, rule in another specification. I did some research, and found the following mentions of LWSP: rfc0733 obs-by rfc0822 rfc0822 defs LWSP-char = SPACE / HTAB obs-by rfc2822 rfc0987 refs rfc0822 rfc1138 refs rfc0822 rfc1148 refs rfc0822 rfc1327 refs rfc0822 rfc1486 refs rfc0822 rfc1505 refs rfc0822 rfc1528 refs rfc0822 rfc1848 defs LWSP-char ::= SPACE / HTAB rfc2017 refs rfc0822 rfc2045 refs rfc0822 rfc2046 refs rfc0822 rfc2110 refs rfc0822 rfc2156 refs rfc0822 rfc2184 refs rfc0822 rfc2231 refs rfc0822 rfc2234 defs LWSP = *(WSP / CRLF WSP) obs-by rfc4234 rfc2243 refs rfc0822 rfc2378 defs LWSP-char = SP | TAB rfc2530 refs rfc2234 rfc2885 defs LWSP = *(WSP / COMMENT / EOL) rfc3015 defs LWSP = *(WSP / COMMENT / EOL) rfc3259 defs LWSP = *(WSP / CRLF WSP) rfc3501 refs rfc2234 rfc3525 defs LWSP = *(WSP / COMMENT / EOL) rfc3875 defs LWSP = SP | HT | NL rfc4234 defs LWSP = *(WSP / CRLF WSP) rfc4646 refs rfc2434 Based on this, I recommend outright deprecation. The RFC4234 definition is wildly different from the RFC822 usage (which is substanitally more often referenced): thus any use of it will tend to confuse. It's also a bit dubious, quite specifically allowing lines which appear to be blank, but aren't. :^( The RFC4234 definition, in fact, is referenced by only 3 RFCs: RFC2530 Indicating Supported Media Features Using Extensions to DSN and MDN RFC3501 INTERNET MESSAGE ACCESS PROTOCOL - VERSION 4rev1 RFC4646 Tags for Identifying Languages The use under RFC2530 is a bit vague (with LWSP wrapping); likewise under RFC3501 (otherwise treat SP as being equivalent to LWSP). The use under RFC4646 has caused known problems. This would seem to justify deprecation, IMHO. YMMV, of course... -- John Leslie [EMAIL PROTECTED] ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: Use of LWSP in ABNF -- consensus call
Lisa Dusseault wrote: 2. The ABNF is a candidate for moving from Draft to Full. Will removing a rule (that is already in use?) or otherwise changing the semantics of the specification, at this point, still permit the document to advance? I had the impression that moving to Full was based on some serious beliefs about a specification's being quite stable. Making this kind of change, this late in the game, would seem to run counter to that. Moving to Internet Standard is indeed something we do carefully, and of course that means investigating proposed changes to make sure they're appropriate, and setting a high bar for accepting them. I believe that's what we're doing here, investigating carefully. I share your concerns about removing rules that are already in use -- that would generally be a bad thing. However I'm interested in the consensus around whether a warning or a deprecation statement would be a good thing. Removing features that have proved to be a Bad Idea has always been listed as one of the possible changes from Proposed to Draft - Draft to Full happens so rarely that I would be hesitant to claim that there's tradition for such changes there. Despite this, I agree with the people who think that a warning comment, rather than removal of the rule, is the Right Way. Harald ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: [ietf-dkim] Re: Use of LWSP in ABNF -- consensus call
Harald Alvestrand wrote: Removing features that have proved to be a Bad Idea has always been listed as one of the possible changes from Proposed to Draft - Draft to Full happens so rarely that I would be hesitant to claim that there's tradition for such changes there. The question is the proved to be criterion. The consensus call was triggered by one documented problem in 10 years. We've had a posting claiming one additional problem (although my own recollection of that bit of history was the the list construct was the issue, rather than LWSP.) So that is a total of at most 2 documented cases in 10-30 years. And keep in mind that the issue is not that the rule does not work but that it is very rarely mis-used. Were we to deprecate every feature in IETF specifications that get mis-implemented a couple of times over 10 years, I suspect much of our technology would be deprecated... d/ -- Dave Crocker Brandenburg InternetWorking bbiw.net ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: Use of LWSP in ABNF -- consensus call
Lisa Dusseault wrote: The issue was initially raised by Frank Hi, a short explanation, initially I hoped that 4234 can be promoted to STD as is. I missed the (now listed) errata in the pending errata mbox. Some months later 4234bis-00 was posted, and if 4234 can't be promoted as is, then that's an opportunity to address this (known) LWSP issue. Just removing it is an idea, but for the reasons stated by Dave I felt that just deprecating it is good enough with less undesirable side-effects. After all it's simple to implement LWSP as specified. But unfortunately it's also simple to destroy critical white space in an apparently empty line. Sorry for the confusion, I should have checked the pending errata mbox before the proposal to promote RFC 4234 to STD. Frank ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: Use of LWSP in ABNF -- consensus call
Lisa Dusseault wrote: I share your concerns about removing rules that are already in use -- that would generally be a bad thing. However I'm interested in the consensus around whether a warning or a deprecation statement would be a good thing. LWSP has a valid meaning and use, and its being misapplied somewhere doesn't make that meaning and usage invalid. I could see a note being added. However, anything more than that is totally inappropriate. Full agreement with Tony here. Ned ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: Use of LWSP in ABNF -- consensus call
--On Tuesday, 15 May, 2007 12:03 -0700 Ned Freed [EMAIL PROTECTED] wrote: Lisa Dusseault wrote: I share your concerns about removing rules that are already in use -- that would generally be a bad thing. However I'm interested in the consensus around whether a warning or a deprecation statement would be a good thing. LWSP has a valid meaning and use, and its being misapplied somewhere doesn't make that meaning and usage invalid. I could see a note being added. However, anything more than that is totally inappropriate. Full agreement with Tony here. +1 ABNF is simply a tool, a grammar, and a set of definitions. I'd almost favor a separate applicability statement that encourages the use of some features and discourages others as appropriate if that is really needed. But a particular element should be removed from the standard only if there is a case to be made that the definition is inadequate or consistent with other parts of the model or grammar. If such inconsistencies actually existed, ABNF should, IMO, be bounced back to Proposed and fixed. Fortunately, that doesn't seem to be the case here, but I would really question what we are doing to ourselves if our grammatical definitions start needing profiles john ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: [ietf-dkim] Re: Use of LWSP in ABNF -- consensus call
On Tue, 15 May 2007, Dave Crocker wrote: So that is a total of at most 2 documented cases in 10-30 years. And keep in mind that the issue is not that the rule does not work but that it is very rarely mis-used. Did you miss my post linking to a description of LWSP-related interop problems in HTTP? http://www.and.org/texts/server-http#implicit-lws Tony. -- f.a.n.finch [EMAIL PROTECTED] http://dotat.at/ FORTH TYNE DOGGER: WEST 4 OR 5, BECOMING VARIABLE 2, THEN SOUTHERLY 3 OR 4. SLIGHT OR MODERATE. SHOWERS THEN MAINLY FAIR. GOOD OCCASIONALLY MODERATE AT FIRST. ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: [ietf-dkim] Re: Use of LWSP in ABNF -- consensus call
Tony Finch wrote: On Tue, 15 May 2007, Dave Crocker wrote: So that is a total of at most 2 documented cases in 10-30 years. And keep in mind that the issue is not that the rule does not work but that it is very rarely mis-used. Did you miss my post linking to a description of LWSP-related interop problems in HTTP? http://www.and.org/texts/server-http#implicit-lws No. d/ -- Dave Crocker Brandenburg InternetWorking bbiw.net ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: Use of LWSP in ABNF -- consensus call
On May 15, 2007, at 10:16 AM, John Leslie wrote: I did some research, and found the following mentions of LWSP: rfc0733 obs-by rfc0822 rfc0822 defs LWSP-char = SPACE / HTAB obs-by rfc2822 rfc0987 refs rfc0822 rfc1138 refs rfc0822 rfc1148 refs rfc0822 rfc1327 refs rfc0822 rfc1486 refs rfc0822 rfc1505 refs rfc0822 rfc1528 refs rfc0822 rfc1848 defs LWSP-char ::= SPACE / HTAB rfc2017 refs rfc0822 rfc2045 refs rfc0822 rfc2046 refs rfc0822 rfc2110 refs rfc0822 rfc2156 refs rfc0822 rfc2184 refs rfc0822 rfc2231 refs rfc0822 rfc2234 defs LWSP = *(WSP / CRLF WSP) obs-by rfc4234 rfc2243 refs rfc0822 rfc2378 defs LWSP-char = SP | TAB rfc2530 refs rfc2234 rfc2885 defs LWSP = *(WSP / COMMENT / EOL) rfc3015 defs LWSP = *(WSP / COMMENT / EOL) rfc3259 defs LWSP = *(WSP / CRLF WSP) rfc3501 refs rfc2234 rfc3525 defs LWSP = *(WSP / COMMENT / EOL) rfc3875 defs LWSP = SP | HT | NL rfc4234 defs LWSP = *(WSP / CRLF WSP) rfc4646 refs rfc2434 Based on this, I recommend outright deprecation. The RFC4234 definition is wildly different from the RFC822 usage (which is substanitally more often referenced): thus any use of it will tend to confuse. It's also a bit dubious, quite specifically allowing lines which appear to be blank, but aren't. :^( The RFC4234 definition, in fact, is referenced by only 3 RFCs: RFC2530 Indicating Supported Media Features Using Extensions to DSN and MDN RFC3501 INTERNET MESSAGE ACCESS PROTOCOL - VERSION 4rev1 RFC4646 Tags for Identifying Languages The use under RFC2530 is a bit vague (with LWSP wrapping); likewise under RFC3501 (otherwise treat SP as being equivalent to LWSP). The use under RFC4646 has caused known problems. This would seem to justify deprecation, IMHO. Agreed. From a standards standpoint, half a dozen definitions for an ABNF mnemonic is absurd. Perhaps something like: The LWSP mnemonic has been deprecated and should not be used in future drafts. Explicit definitions based upon different mnemonics should be used instead. If possible, syntax should guard against possible security concerns related to visual deceptions. -Doug ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: Use of LWSP in ABNF -- consensus call
Lisa Dusseault wrote: The IESG reviewed http://www.ietf.org/internet-drafts/draft-crocker-rfc4234bis-00.txt for publication as Internet Standard and would like to know if there is consensus to recommend against the use of LWSP in future specifications, as it has caused problems recently in DKIM and could cause problems in other places. Some discussion on this point already: - http://www1.ietf.org/mail-archive/web/ietf/current/msg46048.html - http://www1.ietf.org/mail-archive/web/discuss/current/msg00463.html - http://mipassoc.org/pipermail/ietf-dkim/2007q1/007295.html - https://datatracker.ietf.org/public/pidtracker.cgi?command=view_commentid=66440 (in this tracker comment, Chris Newman recommended to remove LWSP, but for backward-compatibility it's probably better to keep it and recommend against use) I agree that LWSP can be problematic. As the LWSP rule only appears in appendix B, the best approach IMHO would be to either leave it in, and have a warning explaining the potential problems close to it, or remove it. The latter sounds simpler, but could cause spec writers that use ABNF to just copy the LWSP rule from RFC4234, ignoring the potential issues with it. The proposed solution to warn about it in the front matter doesn't seem to be a good idea due to locality reasons. If there are reasons to avoud LWSP, they should be stated close to the definition. If this means we need a new revision of the spec, and last-call it again, so be it. No reason to hurry. Best regards, Julian ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: Use of LWSP in ABNF -- consensus call
Lisa Dusseault wrote: The IESG reviewed http://www.ietf.org/internet-drafts/draft-crocker-rfc4234bis-00.txt for publication as Internet Standard and would like to know if there is consensus to recommend against the use of LWSP in future specifications, Lisa, Process: 1 This issue was initially raised in the IESG by Chris Newman, who changed his Discuss, with a statement that he recommended inserting a comment, along the lines that others are also recommending. Unless I've misread the record, all other votes on advancing ABNF from Draft to Full are positive or neutral,. except for your own Discuss. Is this correct? 2. The ABNF is a candidate for moving from Draft to Full. Will removing a rule (that is already in use?) or otherwise changing the semantics of the specification, at this point, still permit the document to advance? I had the impression that moving to Full was based on some serious beliefs about a specification's being quite stable. Making this kind of change, this late in the game, would seem to run counter to that. as it has caused problems recently in DKIM and could cause problems in other places. Semantics: Could cause problems in other places... The DKIM hiccup was the first one I'd heard about. By contrast, linear-white-space was defined in RFC733, in 1977, with RFC822 retaining that definition. It is defined in those places as essentially the same as LWSP in the current ABNF Draft Standard specification. This seems to be one LWSP problem in 30 years. Is this really a sufficient basis for changing the semantics of something that has been stable and widely used for 10-30 years (depending upon how you count)? Are we reasonably comfortable that making the change will not create new and different problems, such as for other uses of ABNF? d/ -- Dave Crocker Brandenburg InternetWorking bbiw.net ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: Use of LWSP in ABNF -- consensus call
On May 14, 2007, at 3:55 PM, Dave Crocker wrote: Lisa Dusseault wrote: The IESG reviewed http://www.ietf.org/internet-drafts/draft- crocker-rfc4234bis-00.txt for publication as Internet Standard and would like to know if there is consensus to recommend against the use of LWSP in future specifications, 1 This issue was initially raised in the IESG by Chris Newman, who changed his Discuss, with a statement that he recommended inserting a comment, along the lines that others are also recommending. Unless I've misread the record, all other votes on advancing ABNF from Draft to Full are positive or neutral,. except for your own Discuss. Is this correct? The issue was initially raised by Frank Ellerman or by various in the DKIM WG depending on how you look at it -- Frank explicitly suggested possible changes to the draft, in his posting to the IETF list. You're right about the voting situation but here's the background: I took on the DISCUSS myself as a placeholder for an issue that the IESG had consensus to investigate further (consensus to investigate what the consensus is). I could have asked somebody else to hold the DISCUSS but this seemed most convenient as long as the rest of the IESG trusted me to investigate. 2. The ABNF is a candidate for moving from Draft to Full. Will removing a rule (that is already in use?) or otherwise changing the semantics of the specification, at this point, still permit the document to advance? I had the impression that moving to Full was based on some serious beliefs about a specification's being quite stable. Making this kind of change, this late in the game, would seem to run counter to that. Moving to Internet Standard is indeed something we do carefully, and of course that means investigating proposed changes to make sure they're appropriate, and setting a high bar for accepting them. I believe that's what we're doing here, investigating carefully. I share your concerns about removing rules that are already in use -- that would generally be a bad thing. However I'm interested in the consensus around whether a warning or a deprecation statement would be a good thing. Thanks, Lisa ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: Use of LWSP in ABNF -- consensus call
Lisa Dusseault wrote: I share your concerns about removing rules that are already in use -- that would generally be a bad thing. However I'm interested in the consensus around whether a warning or a deprecation statement would be a good thing. LWSP has a valid meaning and use, and its being misapplied somewhere doesn't make that meaning and usage invalid. I could see a note being added. However, anything more than that is totally inappropriate. Tony Hansen [EMAIL PROTECTED] ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
