Re: [sqlite] Problem using ICU

2014-04-04 Thread Stadin, Benjamin
Answering to my own question. After some lengthy debugging session I
finally figured that
it’s a problem with the ICU library which I build with SQLite for iOS.
The problem was related to ICU’s obscure loading mechanism.

Ben


Am 03.04.14 19:37 schrieb "Stadin, Benjamin" unter
:

>I¹m having problems to return results from a FTS4 table using ICU.
>
>I¹m doing the usual routine to load an ICU collation:
>
>1) Directly after opening the DB:
>
>SELECT icu_load_collation('de_DE', 'LOCALIZED');
>
>2) Creating some table
>
>CREATE VIRTUAL TABLE ¹sometable' USING fts4 (tokenize='icu' 'LOCALIZED',
>status, firstName, lastName)
>
>
>3) Then inserting some Date via prepared statement and binding params
>
>4) List all entries
>
>SELECT * FROM sometable
>
>So far, so good. The collation loads ok, and all entries in the table are
>listed. 
>
>5) But when I use a MATCH statement, I just get no results back from the
>table:
>
>SELECT * FROM sometable WHERE sometable MATCH '(lastname:a*) OR
>(firstname:a*)' ORDER BY lastname, firstname COLLATE LOCALIZED
>
>
>
>It is definitely related to the CREATE table with "tokenize='icu'
>LOCALIZEDŒ³. When I create the table without this, I get the expected
>results from the very same query. But I¹m completely stuck here, because I
>just get an empty result set and don¹t see any errors at all.
>
>I should mention that I target iOS and compile in ICU with my build of
>SQLite. There is one place in SQLite which I changed like to enable
>loading an ICU data file on the iPhone like so:
>
>SQLITE_PRIVATE int sqlite3IcuInit(sqlite3 *db){
>  struct IcuScalar {
>const char *zName;/* Function name */
>int nArg; /* Number of arguments */
>int enc;  /* Optimal text encoding */
>void *pContext;   /* sqlite3_user_data()
>context */
>void (*xFunc)(sqlite3_context*,int,sqlite3_value**);
>  } scalars[] = {
>{"regexp", 2, SQLITE_ANY,  0, icuRegexpFunc},
>
>{"lower",  1, SQLITE_UTF16,0, icuCaseFunc16},
>{"lower",  2, SQLITE_UTF16,0, icuCaseFunc16},
>{"upper",  1, SQLITE_UTF16, (void*)1, icuCaseFunc16},
>{"upper",  2, SQLITE_UTF16, (void*)1, icuCaseFunc16},
>
>{"lower",  1, SQLITE_UTF8, 0, icuCaseFunc16},
>{"lower",  2, SQLITE_UTF8, 0, icuCaseFunc16},
>{"upper",  1, SQLITE_UTF8,  (void*)1, icuCaseFunc16},
>{"upper",  2, SQLITE_UTF8,  (void*)1, icuCaseFunc16},
>
>{"like",   2, SQLITE_UTF8, 0, icuLikeFunc},
>{"like",   3, SQLITE_UTF8, 0, icuLikeFunc},
>
>{"icu_load_collation",  2, SQLITE_UTF8, (void*)db, icuLoadCollation},
>  };
>
>  // begin custom ICU lib stuff
>
>  // app dir is specified here because i'll avoid to have any calls to
>system icu data dir,
>  // which i think would happen otherwise
>(http://userguide.icu-project.org/icudata)
>  const char *icuDatPath = getPathForICU();
>  u_setDataDirectory(icuDatPath);
>
>  const char *icuBuf = icuData();
>  if (icuBuf != NULL) {
>  UErrorCode err = U_ZERO_ERROR;
>  udata_setAppData_53(getPathForICUCommonDataFile(), &icuBuf, &err);
>  }
>
>  // end custom ICU stuff
>
>
>  int rc = SQLITE_OK;
>  int i;
>
>  for(i=0; rc==SQLITE_OK && i<(int)(sizeof(scalars)/sizeof(scalars[0]));
>i++){
>struct IcuScalar *p = &scalars[i];
>rc = sqlite3_create_function(
>db, p->zName, p->nArg, p->enc, p->pContext, p->xFunc, 0, 0
>);
>  }
>
>  return rc;
>}
>
>
>Any pointers greatly appreciated.
>
>Regards
>Ben
>
>___
>sqlite-users mailing list
>sqlite-users@sqlite.org
>http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


[sqlite] Problem using ICU

2014-04-03 Thread Stadin, Benjamin
I¹m having problems to return results from a FTS4 table using ICU.

I¹m doing the usual routine to load an ICU collation:

1) Directly after opening the DB:

SELECT icu_load_collation('de_DE', 'LOCALIZED');

2) Creating some table

CREATE VIRTUAL TABLE ¹sometable' USING fts4 (tokenize='icu' 'LOCALIZED',
status, firstName, lastName)


3) Then inserting some Date via prepared statement and binding params

4) List all entries

SELECT * FROM sometable

So far, so good. The collation loads ok, and all entries in the table are
listed. 

5) But when I use a MATCH statement, I just get no results back from the
table:

SELECT * FROM sometable WHERE sometable MATCH '(lastname:a*) OR
(firstname:a*)' ORDER BY lastname, firstname COLLATE LOCALIZED



It is definitely related to the CREATE table with "tokenize='icu'
LOCALIZEDŒ³. When I create the table without this, I get the expected
results from the very same query. But I¹m completely stuck here, because I
just get an empty result set and don¹t see any errors at all.

I should mention that I target iOS and compile in ICU with my build of
SQLite. There is one place in SQLite which I changed like to enable
loading an ICU data file on the iPhone like so:

SQLITE_PRIVATE int sqlite3IcuInit(sqlite3 *db){
  struct IcuScalar {
const char *zName;/* Function name */
int nArg; /* Number of arguments */
int enc;  /* Optimal text encoding */
void *pContext;   /* sqlite3_user_data()
context */
void (*xFunc)(sqlite3_context*,int,sqlite3_value**);
  } scalars[] = {
{"regexp", 2, SQLITE_ANY,  0, icuRegexpFunc},

{"lower",  1, SQLITE_UTF16,0, icuCaseFunc16},
{"lower",  2, SQLITE_UTF16,0, icuCaseFunc16},
{"upper",  1, SQLITE_UTF16, (void*)1, icuCaseFunc16},
{"upper",  2, SQLITE_UTF16, (void*)1, icuCaseFunc16},

{"lower",  1, SQLITE_UTF8, 0, icuCaseFunc16},
{"lower",  2, SQLITE_UTF8, 0, icuCaseFunc16},
{"upper",  1, SQLITE_UTF8,  (void*)1, icuCaseFunc16},
{"upper",  2, SQLITE_UTF8,  (void*)1, icuCaseFunc16},

{"like",   2, SQLITE_UTF8, 0, icuLikeFunc},
{"like",   3, SQLITE_UTF8, 0, icuLikeFunc},

{"icu_load_collation",  2, SQLITE_UTF8, (void*)db, icuLoadCollation},
  };

  // begin custom ICU lib stuff

  // app dir is specified here because i'll avoid to have any calls to
system icu data dir,
  // which i think would happen otherwise
(http://userguide.icu-project.org/icudata)
  const char *icuDatPath = getPathForICU();
  u_setDataDirectory(icuDatPath);

  const char *icuBuf = icuData();
  if (icuBuf != NULL) {
  UErrorCode err = U_ZERO_ERROR;
  udata_setAppData_53(getPathForICUCommonDataFile(), &icuBuf, &err);
  }

  // end custom ICU stuff


  int rc = SQLITE_OK;
  int i;

  for(i=0; rc==SQLITE_OK && i<(int)(sizeof(scalars)/sizeof(scalars[0]));
i++){
struct IcuScalar *p = &scalars[i];
rc = sqlite3_create_function(
db, p->zName, p->nArg, p->enc, p->pContext, p->xFunc, 0, 0
);
  }

  return rc;
}


Any pointers greatly appreciated.

Regards
Ben

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users