> UniCode is not the panacea it's purported to be. No, but you have to give them that it's certainly an improvement.
>> UTF-8 is independent of byte-order. The exact encoding (and byte-order) >> should always either be implicit (in the interface's or format's >> definition) or be marked in some way. > > I don't think there is a way to automatically determine the encoding from > the data itself, Yes, you cannot reliably automatically determine encoding. That's why I said you should *know* what data you deal with. (Automatic determination of encoding is a serious problem in dealing with plain text files, but that need not concern a kernel code translation interface such as the one I have in mind.) > and the only way to determine the byte-order (assuming it's > not UTF-8, not a single character, and is unknown from the context) is to > include the special BOM (Byte Order Mark) character as the first > character > of the string. Yes. > In fact, according to the UniCode spec, if the BOM is not > included and the byte-order is not clear from the context, you're > supposed > to assume big-endian. I don't know about that. But I guess that is the case if you say so. > For file system and similar applications, the interface could just always > assume a specific format (probably either UTF-8 or UTF-16LE). Yes. For example, the (in)famous FAT "long" file names are stored in UTF-16LE. Their length is determined by their ASCIZ ("UTF-16LZ") nature ie they are terminated by a 16-bit word of the value zero. If a file system interface (such as Int21/Int21.71) was to be made Unicode-capable I would probably use UTF-8. (Particularly because of the ASCII compatibility, where only characters >= 80h ("codepage-dependent" so to speak) represent code-points >= U+0080.) > For a > general-purpose interface, though, you should be able to handle all > different kinds of possibilities (including things like "UTF-24" and > "UTF-64"). UTF-24 would be pretty funny. (FAT24 is an actual idea I had. Would work well enough.) Even theoretically, UTF-64 doesn't make a lot of sense: a 24-bit (let alone 32-bit) encoding can already represent more values than are currently reserved for all Unicode code-points. Alignment of each single code-point is no particularly good reason to unnecessarily double (you might speak of "bloat" (-; ) the space required to store any given string. 64-bit alignment of the whole string can still be achieved by storing an unused dword behind the actual string if it contains an odd number of dwords; accesses can be aligned by always accessing a whole qword then selecting the appropriate dword and discarding the other. > Also, even though you're dealing with DOS doesn't necessarily > mean everything will be little-endian -- it depends on the source of the > data. Certain hardware interfaces (like SCSI) are inherently big-endian, > and data downloaded from external sources can be almost anything. Yeah. > Another possibility is what my UNI2ASCI program does, which is accept > strings terminated with a specific character (in my case, the UniCode NUL > character, conceptually similar to ASCIIZ). A general-purpose program > should provide more than one way to define a string's length. I guess specifying the length in bytes is good enough. If you want to provide such an interface NUL-terminated (or CP/M-style dollar-terminated (-; ) strings, write a wrapper function which counts the number of non-NUL bytes/words/tri-bytes/dwords/qwords before passing the string to that interface. For non-UTF-8 Unicode encodings, a number of bytes not divisible by the length of the expected units (2, 3, 4, 8) could just cause an error. Generally speaking, error handling is important. Correct UTF-8 validation isn't pretty though. > If you limit > input to only certain encodings or byte-orders or string/character types, > then it ceases to be "general-purpose". Maybe a general-purpose program > is > not what we're really talking about here, but I think one needs to be > developed. Yes, yes. I don't think a general-purpose translation program is what was initially suggested (correct me though). Regards, Christian Just noticing that this grows quite large. If someone finds this unbearable for this list, please speak up to let me know I should cut down the off-topic stuff on my public mails! ------------------------------------------------------------------------------ Increase Visibility of Your 3D Game App & Earn a Chance To Win $500! Tap into the largest installed PC base & get more eyes on your game by optimizing for Intel(R) Graphics Technology. Get started today with the Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs. http://p.sf.net/sfu/intelisp-dev2dev _______________________________________________ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel