Re: [sqlite] Encoding specs & functions overloading

[Jean-Christophe Deschamps]

The 3.6.18 sqlite3.exe CLI produces the same problem: the internal 
functions below can't be overloaded and trying to do so returns 5.

System is XP Pro x86 SP3.

What can I try next ?


>I see that sqlite3.dll is returning 5 == SQLITE_BUSY for the following
>functions:
>
>upper UTF-8
>lower UTF-8
>like   2-arg  UTF-8
>like   3-arg  UTF-8
>glob   2-arg  UTF-8
>
>If I name the functions x* all works well!




___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Encoding specs & functions overloading

[Jean-Christophe Deschamps]

If I set a breakpoint on this:

 rc = sqlite3_create_function(db, p->zName, p->nArg, p->enc, 
p->pContext, p->xFunc, 0, 0);

I see that sqlite3.dll is returning 5 == SQLITE_BUSY for the following 
functions:

upper UTF-8
lower UTF-8
like   2-arg  UTF-8
like   3-arg  UTF-8
glob   2-arg  UTF-8


If I name the functions x* all works well!



___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Encoding specs & functions overloading

[Jean-Christophe Deschamps]

Update: the problem is in the function registration.


I tried to comment out the UTF-16 registration and the really weird 
thing is that using the following code, only GLOB with 3 arguments gets 
actually registered (along with all 1-arg string functions and the two 
collations).

There must be something obvious, but it escapes me completely right now.

Any hint?

---
SQLITE_PRIVATE int unifuzz_init(
   sqlite3 *db
){
   struct FuncScalar {
 const char *zName;/* Function name */
 int nArg; /* Number of arguments */
 int enc;  /* Optimal text encoding */
 void *pContext;   /* sqlite3_user_data() context */
 void (*xFunc)(sqlite3_context*, int, sqlite3_value**);
   } scalars[] = {
 {"version",   0,  SQLITE_ANY,0, 
versionFunc  },
 {"upper", 1,  SQLITE_UTF8,  (void *) unifuzz_upper, 
caseFunc8},
 {"upper", 1,  SQLITE_UTF16, (void *) unifuzz_upper, 
caseFunc16   },
 {"lower", 1,  SQLITE_UTF8,  (void *) unifuzz_lower, 
caseFunc8},
 {"lower", 1,  SQLITE_UTF16, (void *) unifuzz_lower, 
caseFunc16   },
 {"title", 1,  SQLITE_UTF8,  (void *) unifuzz_title, 
caseFunc8},
 {"title", 1,  SQLITE_UTF16, (void *) unifuzz_title, 
caseFunc16   },
 {"fold",  1,  SQLITE_UTF8,   (void *) unifuzz_fold, 
caseFunc8},
 {"fold",  1,  SQLITE_UTF16,  (void *) unifuzz_fold, 
caseFunc16   },
 {"unaccent",  1,  SQLITE_UTF8,   0, 
unaccFunc8   },
 {"unaccent",  1,  SQLITE_UTF16,  0, 
unaccFunc16  },
 {"proper",1,  SQLITE_UTF8,   0, 
properFunc8  },
 {"proper",1,  SQLITE_UTF16,  0, 
properFunc16 },
 {"like",  2,  SQLITE_UTF8, (void *) , likeFunc8},
//{"like",  2,  SQLITE_UTF16, (void *) , 
likeFunc16   },
 {"like",  3,  SQLITE_UTF8,  (void *) , 
likeFunc8},
//{"like",  3,  SQLITE_UTF16, (void *) , 
likeFunc16   },
 {"glob",  2,  SQLITE_UTF8,  (void *) , 
likeFunc8},
//{"glob",  2,  SQLITE_UTF16, (void *) , 
likeFunc16   },
 {"glob",  3,  SQLITE_UTF8,  (void *) , 
likeFunc8},
//{"glob",  3,  SQLITE_UTF16, (void *) , 
likeFunc16   },
 {"typos", 2,  SQLITE_UTF8,   0, 
typosFunc8   },
 {"typos", 2,  SQLITE_UTF16,  0, typosFunc16  }
   };

 int i;
 for(i = 0; (i < (sizeof(scalars) / sizeof(struct FuncScalar))); i++){
 struct FuncScalar *p = [i];
 sqlite3_create_function(db, p->zName, p->nArg, p->enc, 
p->pContext, p->xFunc, 0, 0);
 }

 /* Also override the default NOCASE case-insensitive collation 
sequence. */
 sqlite3_create_collation(db, "NOCASE",  SQLITE_UTF16LE, 0, 
nocase_collate);
 sqlite3_create_collation(db, "NAMES", SQLITE_UTF16LE, 0, 
letters_collate);
 return SQLITE_OK;
}
-



___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Encoding specs & functions overloading

[Jean-Christophe Deschamps]

Hi Pavel,


>I believe you need to show us your sql query. Maybe something in it
>forces SQLite to use UTF-16 version of the function.

Ummm, I don't kno where the problem is, but _any_ simple select will do 
(for me), e.g.:

An UTF-8 base...

CREATE TABLE "PaysISO" (
   "Nom_Iso" CHAR(43),
   "Code_Iso" CHAR(2));

/* Data "PaysISO" */
insert into "PaysISO" values('AFGHANISTAN', 'AF');
insert into "PaysISO" values('AFRIQUE DU SUD', 'ZA');
insert into "PaysISO" values('ÅLAND, ÎLES', 'AX');
insert into "PaysISO" values('ALBANIE', 'AL');
insert into "PaysISO" values('ALGÉRIE', 'DZ');
insert into "PaysISO" values('ALLEMAGNE', 'DE');
insert into "PaysISO" values('ANDORRE', 'AD');
insert into "PaysISO" values('ANGOLA', 'AO');
insert into "PaysISO" values('ANGUILLA', 'AI');
insert into "PaysISO" values('ANTARCTIQUE', 'AQ');
insert into "PaysISO" values('ANTIGUA ET BARBUDA', 'AG');
insert into "PaysISO" values('ANTILLES NÉERLANDAISES', 'AN');
insert into "PaysISO" values('ARABIE SAOUDITE', 'SA');
insert into "PaysISO" values('ARGENTINE', 'AR');
insert into "PaysISO" values('ARMÉNIE', 'AM');
insert into "PaysISO" values('ARUBA', 'AW');
insert into "PaysISO" values('AUSTRALIE', 'AU');
insert into "PaysISO" values('AUTRICHE', 'AT');
...

select * from PaysISO where Nom_ISO like '%ile%';

RecNo Nom_IsoCode_Iso
- -- 
 1 ÅLAND, ÎLESAX
 2 CAÏMANES, ÎLES KY
 3 CHRISTMAS, ÎLE CX
 4 COCOS (KEELING), ÎLES  CC
 5 COOK, ÎLES CK
 6 FALKLAND, ÎLES (MALVINAS)  FK
 7 FÉROÉ, ÎLESFO
 8 GÉORGIE DU SUD ET LES ÎLES SANDWICH DU SUD GS
 9 ÎLE DE MAN IM
10 ÎLES MINEURES ÉLOIGNÉES DES ÉTATS-UNIS UM
11 ÎLES VIERGES BRITANNIQUES  VG
12 ÎLES VIERGES DES ÉTATS-UNISVI
13 MARIANNES DU NORD, ÎLESMP
14 MARSHALL, ÎLES MH
15 NORFOLK, ÎLE   NF
16 SALOMON, ÎLES  SB
17 SVALBARD ET ÎLE JAN MAYEN  SJ
18 TURKS ET CAÏQUES, ÎLES TC

This is a Unicode (non-ICU) folded, unaccented LIKE:

 {"like",2,  SQLITE_UTF8, (void *) 
, likeFunc8},
 {"like",2,  SQLITE_UTF16,(void *) 
, likeFunc16   },
 {"like",3,  SQLITE_UTF8, (void *) 
, likeFunc8},
 {"like",3,  SQLITE_UTF16,(void *) 
, likeFunc16   },
 {"glob",2,  SQLITE_UTF8, (void *) 
, likeFunc8},
 {"glob",2,  SQLITE_UTF16,(void *) 
, likeFunc16   },
 {"glob",3,  SQLITE_UTF8, (void *) 
, likeFunc8},
 {"glob",3,  SQLITE_UTF16,(void *) 
, likeFunc16   },

The same happens for GLOB with 2-arg (UTF-16 version invoked):
select * from PaysISO where Nom_ISO glob '*ILE*';
(same output).

But I just noticed that glob with 3 args invokes the UTF-8 version:
select * from PaysISO where glob('*ILE*', Nom_ISO, '{');
(same output).

While the 3-args like still calls the UTF-16 version:
select * from PaysISO where like('%ile%', Nom_ISO, '{');
(same output).

In all examples above, I've used either plain ASCII or "weirder" 
accented letters, to further test my code.  Also doing this way shows 
that the native SQLite version is not doing anything.  (When things get 
strange, you start looking behind your shoulders.)

The VC++ debugger pops up each time and shows the caller is sqlite3.dll 
every time, so there is no man-in-the-middle attack against my poor 
extension.  I have only v3.6.18 and no modified/recompiled sqlite3.dll 
here.




___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Encoding specs & functions overloading

[Pavel Ivanov]

> I'm using the 3.6.18 Windows dll downloaded direct from the site.
>
> I just re-checked that.

I believe you need to show us your sql query. Maybe something in it
forces SQLite to use UTF-16 version of the function.

Pavel

On Mon, Oct 12, 2009 at 11:44 PM, Jean-Christophe Deschamps
 wrote:
> Thank you for your fast answer.
>
>
>>I'm surprised by this too.  In fact, I cannot reproduce it.
>
> I'm using the 3.6.18 Windows dll downloaded direct from the site.
>
> I just re-checked that.
>
>
>
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Encoding specs & functions overloading

[Jean-Christophe Deschamps]

Thank you for your fast answer.


>I'm surprised by this too.  In fact, I cannot reproduce it.

I'm using the 3.6.18 Windows dll downloaded direct from the site.

I just re-checked that.




___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Encoding specs & functions overloading

[Jean-Christophe Deschamps]

I'm surprised that if I register overloading functions for LIKE or GLOB 
in both UTF-8 and UTF-16, only the UTF-16 version is called despite the 
fact that the database is UTF-8.

I don't see the same behavior with the other scalar functions (lower, 
upper), which call the UTF-8 version as expected for UTF-8 database.

Are LIKE & GLOB special in this respect that a separate entry point 
registered for UTF-8 input is useless?



___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users