Re: Truncation on single byte to UTF-8 conversion (was: Re: ANNOUNCE: DBD::Oracle 1.17 release candidate)

[Tim Bunce]

On Thu, Dec 29, 2005 at 11:56:27AM +0100, Honza Pazdziora wrote:
> 
>         case  96:                               /* CHAR         */
>                 fbh->disize = fbh->dbsize;
>                 if (CS_IS_UTF8(fbh->csid))
>                     fbh->disize = fbh->dbsize * 4;
>                 fbh->prec   = fbh->disize;
>                 break;
> 
> but on a database with character set EE8ISO8859P2 (single byte) and
> with NLS_LANG=AMERICAN_AMERICA.UTF8, the
> 
>       ORA-24345: A Truncation or null fetch error occurred (DBD ERROR: 
> ORA-01406 error on field 1 of 98, ora_type 1)
> 
> is still there. If I remove the
> 
>       if (CS_IS_UTF8(fbh->csid))
> 
> line and just multiply the disize by four for all CHARs, the fetch works.
> 
> If I understand that CS_IS_UTF8 macro correctly, it compares fbh->csid
> to utf8_csid or to al32utf8_csid. However, for my column (database
> in EE8ISO8859P2, NLS_LANG=AMERICAN_AMERICA.UTF8), the trace shows
> 
>     col  1: dbtype 1, scale 0, prec 80, nullok 0, name ID_FIR
>           : dbsize 20, char_used 0, char_size 20, csid 32, csform 1, disize 80
> 
> where csid 32 seems to be the id of (server's) EE8ISO8859P2, not
> of client's UTF8. So the CS_IS_UTF8(fbh->csid) evaluates to false,
> the buffer is not extended and fetching data from single byte database
> to UTF8 client fails.

> Any idea what should I try next to make CS_IS_UTF8 (and fbh->csid)
> behave like it is supposed to?

I think it only needs to *4 if the _client_ csid is utf8.
Try the attached.

If that works then I think similar logic is needed for NCHAR.
Could you reproduce the problem using an NCHAR field with your client
NCHAR charset set to utf8 and your non-nchar charset set to an 8bit
charset?

Tim.

Index: oci8.c
===================================================================
--- oci8.c      (revision 2342)
+++ oci8.c      (working copy)
@@ -1223,7 +1223,7 @@
                /* FALLTHRU */
        case  96:                               /* CHAR         */
                fbh->disize = fbh->dbsize;
-               if (CS_IS_UTF8(fbh->csid)) 
+               if (CS_IS_UTF8(csid)) 
                    fbh->disize = fbh->dbsize * 4;
                fbh->prec   = fbh->disize;
                break;
@@ -1249,7 +1249,7 @@
                break;
 
        case   8:                               /* LONG         */
-                if (CS_IS_UTF8(fbh->csid)) 
+                if (CS_IS_UTF8(csid)) 
                     fbh->disize = long_readlen * 4;
                 else
                     fbh->disize = long_readlen;