The ambiguity of an initial FEFF was not desirable, but this discussion shows that certain things can't be so easily "fixed" by adding characters at a later stage.
The more time elapsed between encoding of the ambiguous character and the later "fix" the more software, the more data, and the more protocols exist that support the original character, creating backwards compatibility issues. Incidentally, this is totally what I expected when the WJ was proposed, but sentiment in favor of its addition ran high at the time... The ZWNBSP was present in Unicode 1.0 (1991) while the WJ was added in 3.2 (2002), that is about 10 years later. We are now an additional 10 years down the road, and instead of clarifying the issue, the interim result is that WJ has muddied the waters instead. Somewhere here are lessons to be learned. A./ -----Original Message----- >From: Doug Ewell <d...@ewellic.org> >Sent: Aug 5, 2011 8:49 AM >To: unicode@unicode.org >Subject: Re: ZWNBSP vs. WJ (was: How is NBH (U0083) Implemented?) > >Jukka K. Korpela <jkorpela at cs dot tut dot fi> wrote: > >> So? It was, and it still often is, better to use ISO 8859-1 rather >> than Unicode, in situations where there no tangible benefit, or just a >> smal l benefit, from using Unicode. For example, many people are still >> conservative about encodings in e-mail, for good reasons, so they use >> ISO 8859-1 or, as you did in your message, windows-1252. > >A word about my encoding "choices." My first message on Thursday was >sent from my home PC, using Windows Live Mail, and it used UTF-8 because >I configured Windows Live Mail to do so. My second message was sent >from my mobile device, and used Windows-1252. I don't know if there is >a way to tell the device to use UTF-8 for outgoing messages, but I can >say it was not my conscious intent to prefer Windows-1252 over Unicode. > >This message is being sent via a Web interface; I guess we'll find out >what encoding it chooses for me. > >> On the other hand, this isn’t comparable to ZWNBSP vs. WJ. These >> control characters do the same job in text, as per the standard, so >> the practical question is simply which one is better supported. > >ZWNBSP, like WJ, is intended to inhibit breaking between words. Despite >the other (and original) intended use of U+FEFF at the start of a text >as a byte-order mark, there is a pervasive belief that an initial U+FEFF >means the text should be treated as beginning with some kind of space >character. This is silly, since there is no concept of "between words" >at the start of a text, but it is nevertheless the way people perceive >things. > >WJ was introduced to encourage users to separate these two functions. >If users don't adopt it, the problem will never be solved. There are >enough issues in Unicode that cannot be fixed due to stability concerns; >it would be nice to be able to fix this one at least. > >I still question how many real-world texts use either U+FEFF or U+2060 >to achieve this non-breaking behavior. > >> ISO 8859-1 and Unicode perform very different jobs, so that using ISO >> 8859-1, you limit your character repertoire (at least as regards to >> directly representable characters, as opposite to various “escape >> notations”). If you don’t need anything outside the ISO 8859-1, the >> choice used to be very simple, though nowadays it has become a little >> more complicated (as e.g. Google Groups seems to munge ISO 8859-1 data >> in quotations but processes UTF-8 properly) > >UTF-8 has the property of being easily detected and verified as such, >which solves part of the Google Groups problem (inability to detect >which SBCS is being used). The other part of the problem is the >practice of using heuristics to override an explicit charset >declaration, but that is a topic for another day. > >> I won’t make any statements about full compliance, but in Microsoft >> Office Word 2007, U+FEFF alias ZWNBSP does its basic job (inside text) >> in most situations whereas U+2060 alias WJ seems to be not recognized >> at all and appears as some sort of a visible box. So to have a job >> jone, there is not much of a choice. (Word 2007 fails to honor ZWNBSP >> semantics after EN DASH, which is bad, but it does not make it useless >> in other situations.) > >It does always come down to a complaint against Microsoft, doesn't it? >Unfortunately, Yucca is right here: opening Word 2007 and pasting a >snippet of text with embedded ZWNBSP does display correctly, while the >same experiment with embedded WJ shows a .notdef box. This seems to be >a font-coverage problem, amplified by Word's silent overriding of user >font choices—changing the font from the default Calibri to DejaVu Sans >(and optionally back to Calibri) makes the display problem go away, but >of course no user could reasonably be expected to go through that. > >-- >Doug Ewell | Thornton, Colorado, USA | RFC 5645, 4645, UTN #14 >www.ewellic.org | www.facebook.com/doug.ewell | @DougEwell > > > >
