Re: [sqlite] Latin-1 characters cannot be supported for Unicode
There are some unintelligible text in my database. Is there any methods to see their byte sequence? Best Regards, Wang Wei -Original Message- From: sqlite-users-boun...@mailinglists.sqlite.org [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On Behalf Of Hick Gunter Sent: Wednesday, June 15, 2016 4:21 PM To: 'SQLite mailing list' Subject: Re: [sqlite] Latin-1 characters cannot be supported for Unicode The answer is very simple: Do not use ANSI/ISO encoding with SQLite. SQLite expects Unicode. -Ursprüngliche Nachricht- Von: sqlite-users-boun...@mailinglists.sqlite.org [mailto:sqlite-users-boun...@mailinglists.sqlite.org] Im Auftrag von Wang, Wei Gesendet: Mittwoch, 15. Juni 2016 04:44 An: SQLite mailing list Betreff: Re: [sqlite] Latin-1 characters cannot be supported for Unicode Under the ANSI encoding environment, I created a table named TEST_PRODUÇÃO in the database. Then I opened this database with sqlite-tool. I ran the sql statement to query all the tables and found the new created table was shown as TEST_PRODU??O. Also this table could not be queried out using the table name TEST_PRODUÇÃO. It seemed that this issue was caused by encoding mismatch. Best Regards, Wang Wei -Original Message- From: sqlite-users-boun...@mailinglists.sqlite.org [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On Behalf Of Chris Brody Sent: Wednesday, June 08, 2016 4:20 PM To: SQLite mailing list Subject: Re: [sqlite] Latin-1 characters cannot be supported for Unicode Hi Wei Wang, Did you populate the database from the sqlite3 CLI tool, your own C program, or from another language? Do you see this when you create a database from scratch, if you use a database created by another program, or in both cases? If you populated the database from the sqlite3 CLI tool, can you post the commands you used to populate the database? If you populated the database from your own C program, can you post a simple test program that populates the database? If you populated the database from another language, can you post a test snippet that shows how you populated the database along with a pointer to which library you are using? What kind of system, CPU, and operating system(s) do you see this behavior on? It should be no problem for sqlite3 to deal with the Latin-1 characters you are using if you do it right. The trick is that sqlite3 is designed to deal with both UTF-8 and UTF-16 (le or be). SQLite stores which encoding is used in the database. The API allows you to use both UTF-8 and UTF-16 encoding, regardless of which encoding is actually used to store the data. I think this is documented properly in sqlite.org, and I found an excellent writeup (though 5 years old) at: http://www.mimec.org/node/297 I also like the Unicode link from Igor. Chris On Wed, Jun 8, 2016 at 3:49 AM, Wang, Wei wrote: > Thanks for your reply! But I found the Latin-1 encoded characters are > listed in the Unicode chart. http://unicode.org/charts/PDF/U0080.pdf > > > Best Regards, > Wang Wei > > -Original Message- > From: sqlite-users-boun...@mailinglists.sqlite.org > [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On Behalf Of > Igor Tandetnik > Sent: Tuesday, June 07, 2016 10:20 PM > To: sqlite-users@mailinglists.sqlite.org > Subject: Re: [sqlite] Latin-1 characters cannot be supported for > Unicode > > On 6/7/2016 3:43 AM, Wang, Wei wrote: >> I met a problem that was maybe caused by the encoding of SQLite. I inserted >> a item which including some Latin1 characters like Ç and à into a table. >> Then I opened the database with SQLite Developer. After I setting the >> encoding to ANSI, the display and the query result for that table were OK. >> However after I setting the encoding to Unicode, these Latin1 characters >> could not be displayed normally, and could not be queried out. Please see >> the attached pictures for the details. > > A byte sequence containing Latin-1-encoded characters Ç or à is not in > fact a valid byte sequence in any Unicode encoding - neither UTF-8 nor > UTF-16 nor any other. If you want Unicode data in your database, then store > Unicode data, and not ANSI, in your database. > -- > Igor Tandetnik > > ___ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > ___ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Latin-1 characters cannot be supported for Unicode
Under the ANSI encoding environment, I created a table named TEST_PRODUÇÃO in the database. Then I opened this database with sqlite-tool. I ran the sql statement to query all the tables and found the new created table was shown as TEST_PRODU??O. Also this table could not be queried out using the table name TEST_PRODUÇÃO. It seemed that this issue was caused by encoding mismatch. Best Regards, Wang Wei -Original Message- From: sqlite-users-boun...@mailinglists.sqlite.org [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On Behalf Of Chris Brody Sent: Wednesday, June 08, 2016 4:20 PM To: SQLite mailing list Subject: Re: [sqlite] Latin-1 characters cannot be supported for Unicode Hi Wei Wang, Did you populate the database from the sqlite3 CLI tool, your own C program, or from another language? Do you see this when you create a database from scratch, if you use a database created by another program, or in both cases? If you populated the database from the sqlite3 CLI tool, can you post the commands you used to populate the database? If you populated the database from your own C program, can you post a simple test program that populates the database? If you populated the database from another language, can you post a test snippet that shows how you populated the database along with a pointer to which library you are using? What kind of system, CPU, and operating system(s) do you see this behavior on? It should be no problem for sqlite3 to deal with the Latin-1 characters you are using if you do it right. The trick is that sqlite3 is designed to deal with both UTF-8 and UTF-16 (le or be). SQLite stores which encoding is used in the database. The API allows you to use both UTF-8 and UTF-16 encoding, regardless of which encoding is actually used to store the data. I think this is documented properly in sqlite.org, and I found an excellent writeup (though 5 years old) at: http://www.mimec.org/node/297 I also like the Unicode link from Igor. Chris On Wed, Jun 8, 2016 at 3:49 AM, Wang, Wei wrote: > Thanks for your reply! But I found the Latin-1 encoded characters are > listed in the Unicode chart. http://unicode.org/charts/PDF/U0080.pdf > > > Best Regards, > Wang Wei > > -Original Message- > From: sqlite-users-boun...@mailinglists.sqlite.org > [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On Behalf Of > Igor Tandetnik > Sent: Tuesday, June 07, 2016 10:20 PM > To: sqlite-users@mailinglists.sqlite.org > Subject: Re: [sqlite] Latin-1 characters cannot be supported for > Unicode > > On 6/7/2016 3:43 AM, Wang, Wei wrote: >> I met a problem that was maybe caused by the encoding of SQLite. I inserted >> a item which including some Latin1 characters like Ç and à into a table. >> Then I opened the database with SQLite Developer. After I setting the >> encoding to ANSI, the display and the query result for that table were OK. >> However after I setting the encoding to Unicode, these Latin1 characters >> could not be displayed normally, and could not be queried out. Please see >> the attached pictures for the details. > > A byte sequence containing Latin-1-encoded characters Ç or à is not in > fact a valid byte sequence in any Unicode encoding - neither UTF-8 nor > UTF-16 nor any other. If you want Unicode data in your database, then store > Unicode data, and not ANSI, in your database. > -- > Igor Tandetnik > > ___ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > ___ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Latin-1 characters cannot be supported for Unicode
Thanks for your reply! But I found the Latin-1 encoded characters are listed in the Unicode chart. http://unicode.org/charts/PDF/U0080.pdf Best Regards, Wang Wei -Original Message- From: sqlite-users-boun...@mailinglists.sqlite.org [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On Behalf Of Igor Tandetnik Sent: Tuesday, June 07, 2016 10:20 PM To: sqlite-users@mailinglists.sqlite.org Subject: Re: [sqlite] Latin-1 characters cannot be supported for Unicode On 6/7/2016 3:43 AM, Wang, Wei wrote: > I met a problem that was maybe caused by the encoding of SQLite. I inserted a > item which including some Latin1 characters like Ç and à into a table. Then > I opened the database with SQLite Developer. After I setting the encoding to > ANSI, the display and the query result for that table were OK. > However after I setting the encoding to Unicode, these Latin1 characters > could not be displayed normally, and could not be queried out. Please see the > attached pictures for the details. A byte sequence containing Latin-1-encoded characters Ç or à is not in fact a valid byte sequence in any Unicode encoding - neither UTF-8 nor UTF-16 nor any other. If you want Unicode data in your database, then store Unicode data, and not ANSI, in your database. -- Igor Tandetnik ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] Latin-1 characters cannot be supported for Unicode
Hi, I met a problem that was maybe caused by the encoding of SQLite. I inserted a item which including some Latin1 characters like Ç and à into a table. Then I opened the database with SQLite Developer. After I setting the encoding to ANSI, the display and the query result for that table were OK. However after I setting the encoding to Unicode, these Latin1 characters could not be displayed normally, and could not be queried out. Please see the attached pictures for the details. Best Regards, Wang Wei ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] Some Latin characters cannot be handled correctly
Recently I faced a problem that when I opened my database with the SQLite Developer tool, if I set the encoding to UNICODE, I failed to queried out the result which including some Latin characters. But when I set the encoding to ASNI, everything is ok. Please see the attached picture to find the details. I supposed that SQLite or SQLite Developer cannot handle some special characters correctly for UNICODE. Best Regards, Wang Wei ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
