Hello Owen, On Sat, 2004-05-01 at 16:46, Owen Taylor wrote: > On Fri, 2004-04-30 at 08:03, Tim Bunce wrote: > > > "You can use UTF8 and AL32UTF8 by setting NLS_LANG for OCI client > > applications. If you do not need supplementary characters, then it > > does not matter whether you choose UTF8 or AL32UTF8. However, if > > your OCI applications might handle supplementary characters, then > > you need to make a decision. Because UTF8 can require up to three > > bytes for each character, one supplementary character is represented > > in two code points, totalling six bytes. In AL32UTF8, one supplementary > > character is represented in one code point, totalling four bytes." > > > > So the key question is... can we just do SvUTF8_on(sv) on either > > of these kinds of Oracle UTF8 encodings? Seems like the answer is > > yes, from what Jarkko says, because they are both valid UTF8. > > We just need to document the issue. > > No, Oracle's "UTF8" is very much not valid UTF-8. Valid UTF-8 cannot > contain surrogates. If you mark a string like this as UTF-8 neither > the Perl core nor other extension modules will be able to interpret > it correctly. > > (As people have pointed out earlier in the thread, > if you want a standard name for this weird form of encoding, that's > "CESU": http://www.unicode.org/reports/tr26/.) > > You'll need to do a conversion pass before you can mark it as UTF-8. >
Your message comes at a PERFECT time! I just spent about 3 hours coming to that same conclusion empiricly: I made the changes to do what tim had asked (just mark the string as UTF8), and it breaks a bunch of stuff, like the 8bit nchar test, and the long test when column type is LONG. I think I am going to back out (or rather... NOT COMMIT) those changes. leaving the code that inspects the fetched string to see if it ("looks like") utf8 before setting the flag. Thanks for chimming in. Lincoln