Re: [sqlite] Latin-1 characters cannot be supported for Unicode

[Hick Gunter]

That the same character is found in both encodings is no surprise. You need to 
look at the actual sequence of bytes.

Comparing a file containing just the "capital A with diaresis" yields

A 1 Byte sequence 0xC4 in ANSI
A 2 Byte sequence 0xC384 in en_US.UTF8 on a RH5 linux system
A 3 Byte Sequence 0xFFFEC4 when converting 0xC4 to UTF-8 in UltraEdit

If you store the single byte 0xC4 then SQLite will retrieve the single byte 
0xC4. If you change the representation layer to expect 0xFFFEC4 or 0xC384 then 
you will be disappointed.

If you put a cat into a box labeled "cat" and then change the label to "dog", 
will that change what is inside? If you sell the box, will the buyer not 
complain?

-Ursprüngliche Nachricht-
Von: sqlite-users-boun...@mailinglists.sqlite.org 
[mailto:sqlite-users-boun...@mailinglists.sqlite.org] Im Auftrag von Wang, Wei
Gesendet: Mittwoch, 08. Juni 2016 03:49
An: SQLite mailing list 
Betreff: Re: [sqlite] Latin-1 characters cannot be supported for Unicode

Thanks for your reply! But I found the Latin-1 encoded characters are listed in 
the Unicode chart. http://unicode.org/charts/PDF/U0080.pdf


Best Regards,
Wang Wei

-Original Message-
From: sqlite-users-boun...@mailinglists.sqlite.org 
[mailto:sqlite-users-boun...@mailinglists.sqlite.org] On Behalf Of Igor 
Tandetnik
Sent: Tuesday, June 07, 2016 10:20 PM
To: sqlite-users@mailinglists.sqlite.org
Subject: Re: [sqlite] Latin-1 characters cannot be supported for Unicode

On 6/7/2016 3:43 AM, Wang, Wei wrote:
> I met a problem that was maybe caused by the encoding of SQLite. I inserted a 
> item which including some Latin1 characters like Ç and  Ã  into a table. Then 
> I opened the database with SQLite Developer. After I setting the encoding to 
> ANSI, the display and the query result for that table were OK.
> However after I setting the encoding to Unicode, these Latin1 characters 
> could not be displayed normally, and could not be queried out. Please see the 
> attached pictures for the details.

A byte sequence containing Latin-1-encoded characters Ç or à is not in fact a 
valid byte sequence in any Unicode encoding - neither UTF-8 nor
UTF-16 nor any other. If you want Unicode data in your database, then store 
Unicode data, and not ANSI, in your database.
--
Igor Tandetnik

___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


___
 Gunter Hick
Software Engineer
Scientific Games International GmbH
FN 157284 a, HG Wien
Klitschgasse 2-4, A-1130 Vienna, Austria
Tel: +43 1 80100 0
E-Mail: h...@scigames.at

This communication (including any attachments) is intended for the use of the 
intended recipient(s) only and may contain information that is confidential, 
privileged or legally protected. Any unauthorized use or dissemination of this 
communication is strictly prohibited. If you have received this communication 
in error, please immediately notify the sender by return e-mail message and 
delete all copies of the original communication. Thank you for your cooperation.


___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Latin-1 characters cannot be supported for Unicode

[Jean-Christophe Deschamps]

At 09:22 08/06/2016, you wrote:

A 3 Byte Sequence 0xFFFEC4 when converting 0xC4 to UTF-8 in UltraEdit


This 3-byte sequence is neither UTF8 or UTF16 even if the BOM would 
make us believe it is UTF16-LE. UTF16 implies 16-bit encoding units, so 
an odd byte length is impossible.

You probably meant FF FE C4 00 for UTF16-LE.

___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Latin-1 characters cannot be supported for Unicode

[Hick Gunter]

Yes, I missed the trailing 00

-Ursprüngliche Nachricht-
Von: sqlite-users-boun...@mailinglists.sqlite.org 
[mailto:sqlite-users-boun...@mailinglists.sqlite.org] Im Auftrag von 
Jean-Christophe Deschamps
Gesendet: Mittwoch, 08. Juni 2016 09:37
An: SQLite mailing list 
Betreff: Re: [sqlite] Latin-1 characters cannot be supported for Unicode

At 09:22 08/06/2016, you wrote:
>A 3 Byte Sequence 0xFFFEC4 when converting 0xC4 to UTF-8 in UltraEdit

This 3-byte sequence is neither UTF8 or UTF16 even if the BOM would make us 
believe it is UTF16-LE. UTF16 implies 16-bit encoding units, so an odd byte 
length is impossible.
You probably meant FF FE C4 00 for UTF16-LE.

___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


___
 Gunter Hick
Software Engineer
Scientific Games International GmbH
FN 157284 a, HG Wien
Klitschgasse 2-4, A-1130 Vienna, Austria
Tel: +43 1 80100 0
E-Mail: h...@scigames.at

This communication (including any attachments) is intended for the use of the 
intended recipient(s) only and may contain information that is confidential, 
privileged or legally protected. Any unauthorized use or dissemination of this 
communication is strictly prohibited. If you have received this communication 
in error, please immediately notify the sender by return e-mail message and 
delete all copies of the original communication. Thank you for your cooperation.


___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Latin-1 characters cannot be supported for Unicode

[Chris Brody]

Hi Wei Wang,

Did you populate the database from the sqlite3 CLI tool, your own C
program, or from another language?

Do you see this when you create a database from scratch, if you use a
database created by another program, or in both cases?

If you populated the database from the sqlite3 CLI tool, can you post
the commands you used to populate the database?

If you populated the database from your own C program, can you post a
simple test program that populates the database?

If you populated the database from another language, can you post a
test snippet that shows how you populated the database along with a
pointer to which library you are using?

What kind of system, CPU, and operating system(s) do you see this behavior on?

It should be no problem for sqlite3 to deal with the Latin-1
characters you are using if you do it right. The trick is that sqlite3
is designed to deal with both UTF-8 and UTF-16 (le or be). SQLite
stores which encoding is used in the database. The API allows you to
use both UTF-8 and UTF-16 encoding, regardless of which encoding is
actually used to store the data. I think this is documented properly
in sqlite.org, and I found an excellent writeup (though 5 years old)
at: http://www.mimec.org/node/297

I also like the Unicode link from Igor.

Chris

On Wed, Jun 8, 2016 at 3:49 AM, Wang, Wei  wrote:
> Thanks for your reply! But I found the Latin-1 encoded characters are listed 
> in the Unicode chart. http://unicode.org/charts/PDF/U0080.pdf
>
>
> Best Regards,
> Wang Wei
>
> -Original Message-
> From: sqlite-users-boun...@mailinglists.sqlite.org 
> [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On Behalf Of Igor 
> Tandetnik
> Sent: Tuesday, June 07, 2016 10:20 PM
> To: sqlite-users@mailinglists.sqlite.org
> Subject: Re: [sqlite] Latin-1 characters cannot be supported for Unicode
>
> On 6/7/2016 3:43 AM, Wang, Wei wrote:
>> I met a problem that was maybe caused by the encoding of SQLite. I inserted 
>> a item which including some Latin1 characters like Ç and  Ã  into a table. 
>> Then I opened the database with SQLite Developer. After I setting the 
>> encoding to ANSI, the display and the query result for that table were OK.
>> However after I setting the encoding to Unicode, these Latin1 characters 
>> could not be displayed normally, and could not be queried out. Please see 
>> the attached pictures for the details.
>
> A byte sequence containing Latin-1-encoded characters Ç or à is not in fact a 
> valid byte sequence in any Unicode encoding - neither UTF-8 nor
> UTF-16 nor any other. If you want Unicode data in your database, then store 
> Unicode data, and not ANSI, in your database.
> --
> Igor Tandetnik
>
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Maybe A Bug in [fcntlSizeHint] function of SQLite?

[sanhua.zh]

In the source code of [fcntlSizeHint] in SQLite 3.9.2,
Its comment said that "If the database is already nBytes or larger, this 
routine is a no-op.”.
But if SQLITE_MAX_MMAP_SIZE0, there is no judgement about file size.


#if SQLITE_MAX_MMAP_SIZE0
 if( pFile-mmapSizeMax0  nBytepFile-mmapSize ){
  int rc;
  if( pFile-szChunk=0 ){
   if( robust_ftruncate(pFile-h, nByte) ){
storeLastErrno(pFile, errno);
return unixLogError(SQLITE_IOERR_TRUNCATE, "ftruncate", pFile-zPath);
   }
  }


  rc = unixMapfile(pFile, nByte);
  return rc;
 }
#endif


The routinue [robust_ftruncate] will run and the file will be cut off if 
nBytefileSize.
This routine may (or may not) cause some problem in SQLite. But for someone 
hacking the source code of SQLite(What I’m doing), it may be a trap.


/*
** This function is called to handle the SQLITE_FCNTL_SIZE_HINT
** file-control operation. Enlarge the database to nBytes in size
** (rounded up to the next chunk-size). If the database is already
** nBytes or larger, this routine is a no-op.
*/
static int fcntlSizeHint(unixFile *pFile, i64 nByte){
 if( pFile-szChunk0 ){
  i64 nSize;  /* Required file size */
  struct stat buf;   /* Used to hold return values of fstat() */
 
  if( osFstat(pFile-h, buf) ){
   return SQLITE_IOERR_FSTAT;
  }


  nSize = ((nByte+pFile-szChunk-1) / pFile-szChunk) * pFile-szChunk;
  if( nSize(i64)buf.st_size ){


#if defined(HAVE_POSIX_FALLOCATE)  HAVE_POSIX_FALLOCATE
   /* The code below is handling the return value of osFallocate()
   ** correctly. posix_fallocate() is defined to "returns zero on success,
   ** or an error number on failure". See the manpage for details. */
   int err;
   do{
err = osFallocate(pFile-h, buf.st_size, nSize-buf.st_size);
   }while( err==EINTR );
   if( err ) return SQLITE_IOERR_WRITE;
#else
   /* If the OS does not have posix_fallocate(), fake it. Write a
   ** single byte to the last byte in each block that falls entirely
   ** within the extended region. Then, if required, a single byte
   ** at offset (nSize-1), to set the size of the file correctly.
   ** This is a similar technique to that used by glibc on systems
   ** that do not have a real fallocate() call.
   */
   int nBlk = buf.st_blksize; /* File-system block size */
   int nWrite = 0;   /* Number of bytes written by seekAndWrite */
   i64 iWrite; /* Next offset to write to */


   iWrite = ((buf.st_size + 2*nBlk - 1)/nBlk)*nBlk-1;
   assert( iWrite=buf.st_size );
   assert( (iWrite/nBlk)==((buf.st_size+nBlk-1)/nBlk) );
   assert( ((iWrite+1)%nBlk)==0 );
   for(/*no-op*/; iWritenSize; iWrite+=nBlk ){
nWrite = seekAndWrite(pFile, iWrite, "", 1);
if( nWrite!=1 ) return SQLITE_IOERR_WRITE;
   }
   if( nWrite==0 || (nSize%nBlk) ){
nWrite = seekAndWrite(pFile, nSize-1, "", 1);
if( nWrite!=1 ) return SQLITE_IOERR_WRITE;
   }
#endif
  }
 }


#if SQLITE_MAX_MMAP_SIZE0
 if( pFile-mmapSizeMax0  nBytepFile-mmapSize ){
  int rc;
  if( pFile-szChunk=0 ){
   if( robust_ftruncate(pFile-h, nByte) ){
storeLastErrno(pFile, errno);
return unixLogError(SQLITE_IOERR_TRUNCATE, "ftruncate", pFile-zPath);
   }
  }


  rc = unixMapfile(pFile, nByte);
  return rc;
 }
#endif


 return SQLITE_OK;
}
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] sqlite built as Multithread

[Jozef Knotek]

Hi,
I am struggling with building SQLite as Multithread 
(-DSQLITE_THREADSAFE=2).
Is there System.Data.SQLite .dll built as Multithread available anywhere please?
Or are there any easy to understand steps how to build it?

Thanks a lot.
Jozef
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] [BugReport]Data changes cannot actually flush to disk

[Richard Hipp]

On 6/7/16, 刘翔  wrote:
> Dear SQLite developers,
>
> Sqlite version: 3.8.8.3
> Linux version: 3.10.31-ltsi
>
> Problem:
> When update database and power off immediately, it cannnot actually make
> changes to database file in the disk.
> Although the HAVE_FDATASYNC comile-time option is true.
> If manually called sync() after executing a SQL statement, it does flush
> the database file to disk.

This is sounds like a hardware or operating system problem.  If I
understand you correctly - fdatasync() is being called but fdatasync()
is not correctly updating the HDD?  Is that right?

SQLite depends on fdatasync() doing what it is suppose to do.  If
fdatasync() misbehaves, there is nothing SQLite can do about that.

What happens if you compile *without* HAVE_FDATASYNC?  Does it work
correctly then?

>
> I think the OSTRACE log can be helpful, so I toke it.
> Could you help to analyse it and tell me why fdatasync() doesn't work?
> 
> LOCK154 SHARED was NONE(NONE,0) pid=2289 (unix)
> LOCK154 SHARED ok (unix)
> READ15416  24 0
> LOCK154 RESERVED was SHARED(SHARED,1) pid=2289 (unix)
> LOCK154 RESERVED ok (unix)
> OPENX   168 /var/user/aaa/bbb.db-journal 0400102
> OPEN168 /var/user/aaa/bbb.db-journal
> WRITE   168   512   0 0
> WRITE   168 4 512 0
> WRITE   168  1024 516 0
> WRITE   168 41540 0
> LOCK154 EXCLUSIVE was RESERVED(RESERVED,1) pid=2289 (unix)
> LOCK154 EXCLUSIVE ok (unix)
> WRITE   168 41544 0
> WRITE   168  10241548 0
> WRITE   168 42572 0
> READ168 03072 0
> SYNC168
> DIRSYNC /var/user/aaa/bbb.db-journal (have_fullfsync=0 fullsync=0)
> OPENDIR 169 /var/user/aaa
> WRITE   16812   0 0
> SYNC168
> READ154 4  24 0
> WRITE   154  1024   0 0
> WRITE   154  1024   20480 0
> SYNC154
> CLOSE   -1
> UNLOCK  154 1 was 4(4,1) pid=2289 (unix)
> UNLOCK  154 0 was 1(1,1) pid=2289 (unix)
> 
>
>
> Cordially yours
>
> LiuXiang
>
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>


-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] [BugReport]Data changes cannot actually flush to disk

[Simon Slavin]

On 7 Jun 2016, at 8:13am, 刘翔  wrote:

> Problem:
> When update database and power off immediately,

How soon do you turn the power off ?  Two seconds ?  Ten seconds ?  One minute ?

What type of hard disk do you have ?  Is it a rotating disk or a solid state 
drive ?

Is the drive you have the database on the boot drive of the computer ?

If you try to make a database on an external drive (Flash drive ?  External 
rotating drive ?) instead of the drive you were using, do you see the same 
problem ?

Simon.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] [BugReport]Data changes cannot actually flush to disk

[r . a . nagy]

Some times writes that have been cached will not be written to disk. In code we 
use the flush() to insure changes are made. In the OS we need to be sure that 
the system has been shutdown properly.

.02 ends.


Sent from my iPhone

> On Jun 8, 2016, at 7:20 AM, Simon Slavin  wrote:
> 
> 
>> On 7 Jun 2016, at 8:13am, 刘翔  wrote:
>> 
>> Problem:
>> When update database and power off immediately,
> 
> How soon do you turn the power off ?  Two seconds ?  Ten seconds ?  One 
> minute ?
> 
> What type of hard disk do you have ?  Is it a rotating disk or a solid state 
> drive ?
> 
> Is the drive you have the database on the boot drive of the computer ?
> 
> If you try to make a database on an external drive (Flash drive ?  External 
> rotating drive ?) instead of the drive you were using, do you see the same 
> problem ?
> 
> Simon.
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] sqlite built as Multithread

[Joe Mistachkin]

Jozef Knotek
>
> I am struggling with building SQLite as Multithread
> (-DSQLITE_THREADSAFE=2).
>

For various reasons, using -DSQLITE_THREADSAFE=2 with System.Data.SQLite
is not recommended.  Only -DSQLITE_THREADSAFE=1 should be used with native
SQLite libraries that need to work with System.Data.SQLite.

--
Joe Mistachkin

___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users