Re: [sqlite] ?==?utf-8?q? Sqlite 3.31.0 breaks firefox and thunderbird
On Thursday, January 23, 2020 17:00 CET, Richard Hipp wrote: > On 1/23/20, Bernhard Rosenkraenzer wrote: > > Hi, > > after updating sqlite to 3.31.0, both firefox and thunderbird crash on > > startup (rebuilding them against the newer sqlite doesn't help). > > Is this related to https://bugzilla.mozilla.org/show_bug.cgi?id=1607902 Hi, thanks for the pointer - it's indeed caused by that, and the fix mentioned there works (after some tweaking to make it apply on top of current firefox and thunderbird releases). If anyone needs it, here's the fix adapted to current firefox and thunderbird releases. https://github.com/OpenMandrivaAssociation/firefox/blob/master/firefox-72.0.2-sqlite-3.31.patch https://github.com/OpenMandrivaAssociation/thunderbird/blob/master/thunderbird-68.4.1-sqlite-3.31.patch ttyl bero ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] UTF-8
On Wed, Feb 20, 2013 at 3:43 PM, Marco Bambini wrote:
> Sometimes instead of a query like:
> "INSERT INTO foo (col1) VALUES ('Boxhagener Straße');"
>
> I receive the utf-8 encoded counterpart:
> "INSERT INTO foo (col1) VALUES ('Boxhagener Str\u00c3\u009fe');"
>
> Is there a way to automatically decode this latest query to UTF-8 just
> using sqlite API?
> Please note that since I have access to the sqlite amalgamation source
> code I could also expose an internal function.
>
SQLite assumes that all of its inputs are UTF (either UTF8 or UTF16). If
you give it an input of MBCS, it will store what you give it, byte for
byte, but for comparison and lookup purposes it always assumes it is UTF.
SQLite never tries to convert between MBCS and UTF (except as required deep
down inside the windows VFS, but that shouldn't be visible to the
application.)
>
> Thanks.
> --
> Marco Bambini
> http://www.sqlabs.com
> http://twitter.com/sqlabs
> http://instagram.com/sqlabs
>
>
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
--
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] UTF-8 strings & ICU
On 18 Mar 2011, at 5:07am, ashish yadav wrote: > Can you please also provide some code sample for same ? > > i am not able to understand, your point "When a custom collation is > registered, it may indicate whether it wants its strings in UTF-8, UTF-16 or > either." The ICU extension for SQLite does some of its work by defining a custom collation. You don't need to do this yourself, it's already in the code. So it will handle conversion between UTF formats itself. Just handle your strings as you normally would. There doesn't seem to be a page on www.sqlite.org about ICU. It would be nice to have some sample code. The best page so far seems to be http://www.sqlite.org/src/artifact?ci=trunk&filename=ext/icu/README.txt Simon. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] UTF-8 strings & ICU
Hi Igor, Thanks a lot for info. Can you please also provide some code sample for same ? i am not able to understand, your point "When a custom collation is registered, it may indicate whether it wants its strings in UTF-8, UTF-16 or either." Thanks & Regards Ashish On Fri, Mar 18, 2011 at 10:06 AM, Igor Tandetnik wrote: > On 3/18/2011 12:27 AM, ashish yadav wrote: > > My application is using UTF-8 strings ie database contain UTF-8 strings. > > For sorting requirement of application ,i use ICU which is integrated > with > > sqlite3. > > > > Now ICU support UTF-16 encode string for its operation. > > > > Can any one please tell me how to deal with this situation ? > > If ICU/Sqlite3 by default take care of this ? > > Yes. When a custom collation is registered, it may indicate whether it > wants its strings in UTF-8, UTF-16 or either. SQLite will automatically > convert all strings to the form the collation wants, before calling it. > > Bottom line, SQLite and ICU will settle the matter between themselves. > You don't need to do anything special. > -- > Igor Tandetnik > > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] UTF-8 strings & ICU
On 3/18/2011 12:27 AM, ashish yadav wrote: > My application is using UTF-8 strings ie database contain UTF-8 strings. > For sorting requirement of application ,i use ICU which is integrated with > sqlite3. > > Now ICU support UTF-16 encode string for its operation. > > Can any one please tell me how to deal with this situation ? > If ICU/Sqlite3 by default take care of this ? Yes. When a custom collation is registered, it may indicate whether it wants its strings in UTF-8, UTF-16 or either. SQLite will automatically convert all strings to the form the collation wants, before calling it. Bottom line, SQLite and ICU will settle the matter between themselves. You don't need to do anything special. -- Igor Tandetnik ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] UTF-8
Hey,
i found the solution for the utf-8 encoding problem:
i used the:
wchar_t chAppWinTextUnicode[MAX_PATH] = { 0 };
::*GetWindowTextW*(hWnd, chAppWinTextUnicode, MAX_PATH);
int nSize = *WideCharToMultiByte*(/*CP_UTF8*/, 0, chAppWinTextUnicode,
-1, NULL, 0, NULL, NULL);
char* pchBuffer= (char*) malloc(nSize + 1);
*WideCharToMultiByte*(/*CP_UTF8*/, 0, chAppWinTextUnicode, -1,
pchBuffer, nSize, NULL, NULL);
...
...
the save the pchBuffer to std::string str(pchBuffer);
then this string will be correctly saved into sqlite :)
thanks very much for your helps!
best regards
Ming
On 16.12.2010 17:34, Afriza N. Arief wrote:
> @Cory: correct.
>
> Fixed Code:
>
> HWND hWnd = /* from somewhere */;
> int nLen = GetWindowTextLength(hWnd) + 1;
> WCHAR* lpszText = (WCHAR*)malloc(nLen*sizeof(WCHAR));
> // std::wstring str; str.resize(nLen);
> nLen = GetWindowText(hWnd,lpszText,nLen);
> // str.resize(nLen=GetWindowText(hWnd,&str[0],nLen));
> sqlite3_bind_text16(stmt,1,lpszText,nLen*sizeof(WCHAR),SQLITE_TRANSIENT);
> // (.., str.data(),nLen*sizeof(std::wstring::value_type),SQLITE_TRANSIENT);
> free(lpszText); // not needed for std::wstring
>
> When you first time create the database, use sqlite3_open() to store
> the data as utf8. I believe bind_text16() will convert from utf16 to
> utf8 for you.
>
> See:
> - GetWindowTextLength() http://msdn.microsoft.com/en-us/library/ms633521.aspx
> - GetWindowText() http://msdn.microsoft.com/en-us/library/ms633520.aspx
> - sqlite3_bind_text16() http://www.sqlite.org/c3ref/bind_blob.html
>
> Note:
> - You need your project to be compiled in UNICODE
>
> On Fri, Dec 17, 2010 at 12:24 AM, Cory Nelson wrote:
>> Just a quick note -- TCHAR is not always UTF-16, and
>> sqlite3_bind_text16 takes a void* so it will happily take whatever you
>> give it. You should be using wchar_t directly instead of TCHAR, so an
>> error can be caught when you use GetWindowText.
>>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] UTF-8
Hello afriza, thanks for your quick replay. if i use the WCHAR or std'::wstring, should i used the _UNICODE macro in the Visual Studio Project setting? best regards ming On 16.12.2010 17:34, Afriza N. Arief wrote: > @Cory: correct. > > Fixed Code: > > HWND hWnd = /* from somewhere */; > int nLen = GetWindowTextLength(hWnd) + 1; > WCHAR* lpszText = (WCHAR*)malloc(nLen*sizeof(WCHAR)); > // std::wstring str; str.resize(nLen); > nLen = GetWindowText(hWnd,lpszText,nLen); > // str.resize(nLen=GetWindowText(hWnd,&str[0],nLen)); > sqlite3_bind_text16(stmt,1,lpszText,nLen*sizeof(WCHAR),SQLITE_TRANSIENT); > // (.., str.data(),nLen*sizeof(std::wstring::value_type),SQLITE_TRANSIENT); > free(lpszText); // not needed for std::wstring > > When you first time create the database, use sqlite3_open() to store > the data as utf8. I believe bind_text16() will convert from utf16 to > utf8 for you. > > See: > - GetWindowTextLength() http://msdn.microsoft.com/en-us/library/ms633521.aspx > - GetWindowText() http://msdn.microsoft.com/en-us/library/ms633520.aspx > - sqlite3_bind_text16() http://www.sqlite.org/c3ref/bind_blob.html > > Note: > - You need your project to be compiled in UNICODE > > On Fri, Dec 17, 2010 at 12:24 AM, Cory Nelson wrote: >> Just a quick note -- TCHAR is not always UTF-16, and >> sqlite3_bind_text16 takes a void* so it will happily take whatever you >> give it. You should be using wchar_t directly instead of TCHAR, so an >> error can be caught when you use GetWindowText. >> > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] UTF-8
@Cory: correct. Fixed Code: HWND hWnd = /* from somewhere */; int nLen = GetWindowTextLength(hWnd) + 1; WCHAR* lpszText = (WCHAR*)malloc(nLen*sizeof(WCHAR)); // std::wstring str; str.resize(nLen); nLen = GetWindowText(hWnd,lpszText,nLen); // str.resize(nLen=GetWindowText(hWnd,&str[0],nLen)); sqlite3_bind_text16(stmt,1,lpszText,nLen*sizeof(WCHAR),SQLITE_TRANSIENT); // (.., str.data(),nLen*sizeof(std::wstring::value_type),SQLITE_TRANSIENT); free(lpszText); // not needed for std::wstring When you first time create the database, use sqlite3_open() to store the data as utf8. I believe bind_text16() will convert from utf16 to utf8 for you. See: - GetWindowTextLength() http://msdn.microsoft.com/en-us/library/ms633521.aspx - GetWindowText() http://msdn.microsoft.com/en-us/library/ms633520.aspx - sqlite3_bind_text16() http://www.sqlite.org/c3ref/bind_blob.html Note: - You need your project to be compiled in UNICODE On Fri, Dec 17, 2010 at 12:24 AM, Cory Nelson wrote: > > Just a quick note -- TCHAR is not always UTF-16, and > sqlite3_bind_text16 takes a void* so it will happily take whatever you > give it. You should be using wchar_t directly instead of TCHAR, so an > error can be caught when you use GetWindowText. > ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] UTF-8
Just a quick note -- TCHAR is not always UTF-16, and sqlite3_bind_text16 takes a void* so it will happily take whatever you give it. You should be using wchar_t directly instead of TCHAR, so an error can be caught when you use GetWindowText. -- Cory Nelson http://int64.org On Thu, Dec 16, 2010 at 8:08 AM, Afriza N. Arief wrote: > Try this: > > HWND hWnd = /* from somewhere */; > int nLen = GetWindowTextLength(hWnd) + 1; > TCHAR* lpszText = (TCHAR*)malloc(nLen*sizeof(TCHAR)); // std::wstring str; > str.resize(nLen); > nLen = GetWindowText(hWnd,lpszText,nLen); // > str.resize(nLen=GetWindowText(hWnd,&str[0],nLen)); > > sqlite3_bind_text16(stmt,1,lpszText,nLen*sizeof(TCHAR),SQLITE_TRANSIENT); > // (.., str.data(),nLen*sizeof(std::wstring::value_type),SQLITE_TRANSIENT); > free(lpszText); // not needed for std::wstring > > When you first time create the database, use sqlite3_open() to store the > data as utf8. I believe bind_text16() will convert from utf16 to utf8 for > you. > > See: > - GetWindowTextLength() > http://msdn.microsoft.com/en-us/library/ms633521.aspx > - GetWindowText() http://msdn.microsoft.com/en-us/library/ms633520.aspx > - sqlite3_bind_text16() http://www.sqlite.org/c3ref/bind_blob.html > > > On Thu, Dec 16, 2010 at 10:51 PM, Pavel Ivanov wrote: > >> > std::string strText = GetWindowsTitle(...); >> > the GetWindowsTitle occupied sometime with the umlaut. >> > so how will you do, if you want save std::string into sqlite with keeped >> > umlaut? >> >> >> Pavel >> >> On Thu, Dec 16, 2010 at 8:40 AM, Ming Lu wrote: >> > i tried again with the firefox plugin works very well with the umlaut.(i >> > update the feld with a "ä", it can been shown correctly). >> > >> > in my code: >> > >> > std::string strText = GetWindowsTitle(...); >> > >> > the GetWindowsTitle occupied sometime with the umlaut. >> > >> > so how will you do, if you want save std::string into sqlite with keeped >> > umlaut? >> > >> > >> > On 16.12.2010 14:26, Martin Engelschalk wrote: >> >> Hello Ming, >> >> >> >> sqlite does nothing to transform data between codepages, and it assumes >> >> that data you insert is passed in UTF8. >> >> However, sqlite will acept any data and store it. >> >> >> >> If the firefox plugin does not show you data correctly, then you >> >> problably did not pass correct UTF8 to sqlite. Can you check this? >> >> >> >> I just checked the firefox (0.6.5) plugin with my databases, it works >> >> correctly for me. >> >> >> ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] UTF-8
Try this: HWND hWnd = /* from somewhere */; int nLen = GetWindowTextLength(hWnd) + 1; TCHAR* lpszText = (TCHAR*)malloc(nLen*sizeof(TCHAR)); // std::wstring str; str.resize(nLen); nLen = GetWindowText(hWnd,lpszText,nLen); // str.resize(nLen=GetWindowText(hWnd,&str[0],nLen)); sqlite3_bind_text16(stmt,1,lpszText,nLen*sizeof(TCHAR),SQLITE_TRANSIENT); // (.., str.data(),nLen*sizeof(std::wstring::value_type),SQLITE_TRANSIENT); free(lpszText); // not needed for std::wstring When you first time create the database, use sqlite3_open() to store the data as utf8. I believe bind_text16() will convert from utf16 to utf8 for you. See: - GetWindowTextLength() http://msdn.microsoft.com/en-us/library/ms633521.aspx - GetWindowText() http://msdn.microsoft.com/en-us/library/ms633520.aspx - sqlite3_bind_text16() http://www.sqlite.org/c3ref/bind_blob.html On Thu, Dec 16, 2010 at 10:51 PM, Pavel Ivanov wrote: > > std::string strText = GetWindowsTitle(...); > > the GetWindowsTitle occupied sometime with the umlaut. > > so how will you do, if you want save std::string into sqlite with keeped > > umlaut? > > > Pavel > > On Thu, Dec 16, 2010 at 8:40 AM, Ming Lu wrote: > > i tried again with the firefox plugin works very well with the umlaut.(i > > update the feld with a "ä", it can been shown correctly). > > > > in my code: > > > > std::string strText = GetWindowsTitle(...); > > > > the GetWindowsTitle occupied sometime with the umlaut. > > > > so how will you do, if you want save std::string into sqlite with keeped > > umlaut? > > > > > > On 16.12.2010 14:26, Martin Engelschalk wrote: > >> Hello Ming, > >> > >> sqlite does nothing to transform data between codepages, and it assumes > >> that data you insert is passed in UTF8. > >> However, sqlite will acept any data and store it. > >> > >> If the firefox plugin does not show you data correctly, then you > >> problably did not pass correct UTF8 to sqlite. Can you check this? > >> > >> I just checked the firefox (0.6.5) plugin with my databases, it works > >> correctly for me. > >> > ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] UTF-8
> std::string strText = GetWindowsTitle(...); > the GetWindowsTitle occupied sometime with the umlaut. > so how will you do, if you want save std::string into sqlite with keeped > umlaut? My bet would be that GetWindowsTitle (even if it's not GetWindowTitle from Win32 API) returns string to you in your default encoding (which probably is Windows-1252). So before feeding that std::string into SQLite you need to change its encoding to UTF-8. Probably the best way of such conversion is to use some table mapping Windows-1252 bytes to UTF-8 sequences and convert your string char-by-char. Pavel On Thu, Dec 16, 2010 at 8:40 AM, Ming Lu wrote: > Hello Martin, > > thank you very much for your suggestion. > > i tried again with the firefox plugin works very well with the umlaut.(i > update the feld with a "ä", it can been shown correctly). > > in my code: > > std::string strText = GetWindowsTitle(...); > > the GetWindowsTitle occupied sometime with the umlaut. > > so how will you do, if you want save std::string into sqlite with keeped > umlaut? > > thanks a lot > best regards > ming > > On 16.12.2010 14:26, Martin Engelschalk wrote: >> Hello Ming, >> >> sqlite does nothing to transform data between codepages, and it assumes >> that data you insert is passed in UTF8. >> However, sqlite will acept any data and store it. >> >> If the firefox plugin does not show you data correctly, then you >> problably did not pass correct UTF8 to sqlite. Can you check this? >> >> I just checked the firefox (0.6.5) plugin with my databases, it works >> correctly for me. >> >> Pictures / Attachments do not make it to the list. >> >> Martin >> >> >> Am 16.12.2010 14:17, schrieb Ming Lu: >>> Hello everyone, >>> >>> i am faceing a problem with unicode to save german umlaute in sqlite: >>> >>> here ist the problem: >>> >>> dev-envoriment: >>> Visual studio 2008 >>> SQLite 3.6.20 >>> >>> i used the sqlite c/c++ interface and open a db used sqlite3_open. >>> during the running of my application will the german-umlaut or other >>> symbol transported into the database. i use the SQLite Manager(firefox >>> plugin) to open the db and the umlaut cannot be correctly saved into the >>> database(see the pic) : >>> >>> -> it should be "ä". >>> >>> my question is, how can use the sqlite c/c++ interface to save the >>> unicode correctly into a sqlite database? >>> >>> thanks a lot >>> best regards >>> >>> Ming >>> ___ >>> sqlite-users mailing list >>> sqlite-users@sqlite.org >>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users >> ___ >> sqlite-users mailing list >> sqlite-users@sqlite.org >> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] UTF-8
I suggest you try using wchar_t* or std::wstring as std::string is not Unicode afaik. Regards, Jonas On Thu, Dec 16, 2010 at 2:40 PM, Ming Lu wrote: > Hello Martin, > > thank you very much for your suggestion. > > i tried again with the firefox plugin works very well with the umlaut.(i > update the feld with a "ä", it can been shown correctly). > > in my code: > > std::string strText = GetWindowsTitle(...); > > the GetWindowsTitle occupied sometime with the umlaut. > > so how will you do, if you want save std::string into sqlite with keeped > umlaut? > > thanks a lot > best regards > ming > > On 16.12.2010 14:26, Martin Engelschalk wrote: >> Hello Ming, >> >> sqlite does nothing to transform data between codepages, and it assumes >> that data you insert is passed in UTF8. >> However, sqlite will acept any data and store it. >> >> If the firefox plugin does not show you data correctly, then you >> problably did not pass correct UTF8 to sqlite. Can you check this? >> >> I just checked the firefox (0.6.5) plugin with my databases, it works >> correctly for me. >> >> Pictures / Attachments do not make it to the list. >> >> Martin >> >> >> Am 16.12.2010 14:17, schrieb Ming Lu: >>> Hello everyone, >>> >>> i am faceing a problem with unicode to save german umlaute in sqlite: >>> >>> here ist the problem: >>> >>> dev-envoriment: >>> Visual studio 2008 >>> SQLite 3.6.20 >>> >>> i used the sqlite c/c++ interface and open a db used sqlite3_open. >>> during the running of my application will the german-umlaut or other >>> symbol transported into the database. i use the SQLite Manager(firefox >>> plugin) to open the db and the umlaut cannot be correctly saved into the >>> database(see the pic) : >>> >>> -> it should be "ä". >>> >>> my question is, how can use the sqlite c/c++ interface to save the >>> unicode correctly into a sqlite database? >>> >>> thanks a lot >>> best regards >>> >>> Ming >>> ___ >>> sqlite-users mailing list >>> sqlite-users@sqlite.org >>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users >> ___ >> sqlite-users mailing list >> sqlite-users@sqlite.org >> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] UTF-8
Hello Martin, thank you very much for your suggestion. i tried again with the firefox plugin works very well with the umlaut.(i update the feld with a "ä", it can been shown correctly). in my code: std::string strText = GetWindowsTitle(...); the GetWindowsTitle occupied sometime with the umlaut. so how will you do, if you want save std::string into sqlite with keeped umlaut? thanks a lot best regards ming On 16.12.2010 14:26, Martin Engelschalk wrote: > Hello Ming, > > sqlite does nothing to transform data between codepages, and it assumes > that data you insert is passed in UTF8. > However, sqlite will acept any data and store it. > > If the firefox plugin does not show you data correctly, then you > problably did not pass correct UTF8 to sqlite. Can you check this? > > I just checked the firefox (0.6.5) plugin with my databases, it works > correctly for me. > > Pictures / Attachments do not make it to the list. > > Martin > > > Am 16.12.2010 14:17, schrieb Ming Lu: >> Hello everyone, >> >> i am faceing a problem with unicode to save german umlaute in sqlite: >> >> here ist the problem: >> >> dev-envoriment: >> Visual studio 2008 >> SQLite 3.6.20 >> >> i used the sqlite c/c++ interface and open a db used sqlite3_open. >> during the running of my application will the german-umlaut or other >> symbol transported into the database. i use the SQLite Manager(firefox >> plugin) to open the db and the umlaut cannot be correctly saved into the >> database(see the pic) : >> >> -> it should be "ä". >> >> my question is, how can use the sqlite c/c++ interface to save the >> unicode correctly into a sqlite database? >> >> thanks a lot >> best regards >> >> Ming >> ___ >> sqlite-users mailing list >> sqlite-users@sqlite.org >> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] UTF-8
Hello Ming, sqlite does nothing to transform data between codepages, and it assumes that data you insert is passed in UTF8. However, sqlite will acept any data and store it. If the firefox plugin does not show you data correctly, then you problably did not pass correct UTF8 to sqlite. Can you check this? I just checked the firefox (0.6.5) plugin with my databases, it works correctly for me. Pictures / Attachments do not make it to the list. Martin Am 16.12.2010 14:17, schrieb Ming Lu: > Hello everyone, > > i am faceing a problem with unicode to save german umlaute in sqlite: > > here ist the problem: > > dev-envoriment: > Visual studio 2008 > SQLite 3.6.20 > > i used the sqlite c/c++ interface and open a db used sqlite3_open. > during the running of my application will the german-umlaut or other > symbol transported into the database. i use the SQLite Manager(firefox > plugin) to open the db and the umlaut cannot be correctly saved into the > database(see the pic) : > > -> it should be "ä". > > my question is, how can use the sqlite c/c++ interface to save the > unicode correctly into a sqlite database? > > thanks a lot > best regards > > Ming > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] UTF-8 and UTF-16
>From: Jens Miltner >Sent: Wed, February 3, 2010 9:46:06 AM >Just another thought to consider: depending on the amount of non-ASCII >(or non-roman) string data stored in your database, in may be more >efficient to use UTF-8 encoding rather than UTF-16 encoding: >UTF-8 takes up less space for ASCII or roman text, whereas it may take >up more space for other characters. This really depends on the mix. Thank you for your suggestions!. I will go with UTF-8 encoding. Regards, Samuel __ Make your browsing faster, safer, and easier with the new Internet Explorer® 8. Optimized for Yahoo! Get it Now for Free! at http://downloads.yahoo.com/ca/internetexplorer/ ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] UTF-8 and UTF-16
Am 01.02.2010 um 00:02 schrieb a1rex: > I am planning to store text in a database which has to accommodate a > few international languages. > In this case I have to use UTF-16LE encoding for my TEXT fields. > I know that once an encoding has been set for a database, it cannot > be changed. Just another thought to consider: depending on the amount of non-ASCII (or non-roman) string data stored in your database, in may be more efficient to use UTF-8 encoding rather than UTF-16 encoding: UTF-8 takes up less space for ASCII or roman text, whereas it may take up more space for other characters. This really depends on the mix. However, we found that our database performance improved quite a bit when we switched back to UTF-8 (granted, most of the characters in our data can be represented by single-byte UTF-8 sequences), mainly due to the reduced overall database size (probably fitting more columns on a single page in the database, thus requiring less read operations to access a single record). But as I said, this depends a lot on the data you will store, the total size of the database and the structure of your tables, so you'll have to evaluate yourself... ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] UTF-8 and UTF-16
Thank you for your help! This call: sqlite3_exec(handle,"PRAGMA encoding = \"UTF-16\"",NULL,NULL,&zErr); works. Samuel - Original Message From: Igor Tandetnik To: sqlite-users@sqlite.org Sent: Sun, January 31, 2010 6:12:27 PM Subject: Re: [sqlite] UTF-8 and UTF-16 a1rex wrote: > I am planning to store text in a database which has to accommodate a few > international languages. > In this case I have to use UTF-16LE encoding for my TEXT fields. Perhaps there are some external reasons, but there's nothing in SQLite that would force this. For example, UTF-8 is just as capable of representing any Unicode string as UTF-16. > I know that once an encoding has been set for a database, it cannot be > changed. > Do BLOBS are effected? No. > I guess I cannot mix TEXT UTF-8 and UTF-16 columns in the same data base, > can I? Why would you want to? > Can I get away with a database encoded in UTF-8 by using > sqlite3_bind_text16() function for TEXT fields when I need UTF-16 text? Yes. SQLite automatically converts between UTF-8 and UTF-16 as necessary. > After opening empty database, I tried to execute: > > sqlite3_exec(handle,"PRAGMA encoding = UTF-16",NULL,NULL,&zErr); > > But I got the following error: > PRAMA error: near "-": syntax error. What did I do wrong? It's PRAGMA encoding = "UTF-16". The parameter needs to be quoted. Igor Tandetnik ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users __ Yahoo! Canada Toolbar: Search from anywhere on the web, and bookmark your favourite sites. Download it now http://ca.toolbar.yahoo.com. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] UTF-8 and UTF-16
On Sun, 31 Jan 2010 18:12:27 -0500, "Igor Tandetnik" wrote: >It's PRAGMA encoding = "UTF-16". The parameter needs to be quoted. This is indeed the syntax description. Peculiarly, anywhere else, string literals have to be single quoted. Is this something to iron out in v4 ? -- ( Kees Nuyt ) c[_] ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] UTF-8 and UTF-16
a1rex wrote: > I am planning to store text in a database which has to accommodate a few > international languages. > In this case I have to use UTF-16LE encoding for my TEXT fields. Perhaps there are some external reasons, but there's nothing in SQLite that would force this. For example, UTF-8 is just as capable of representing any Unicode string as UTF-16. > I know that once an encoding has been set for a database, it cannot be > changed. > Do BLOBS are effected? No. > I guess I cannot mix TEXT UTF-8 and UTF-16 columns in the same data base, > can I? Why would you want to? > Can I get away with a database encoded in UTF-8 by using > sqlite3_bind_text16() function for TEXT fields when I need UTF-16 text? Yes. SQLite automatically converts between UTF-8 and UTF-16 as necessary. > After opening empty database, I tried to execute: > > sqlite3_exec(handle,"PRAGMA encoding = UTF-16",NULL,NULL,&zErr); > > But I got the following error: > PRAMA error: near "-": syntax error. What did I do wrong? It's PRAGMA encoding = "UTF-16". The parameter needs to be quoted. Igor Tandetnik ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] UTF-8 by default?
J. R. Westmoreland wrote: > I haven't found, yet, where it says this but, I assume UTF-8 is the > default storage for character strings in the database? http://sqlite.org/c3ref/open.html The default encoding for the database will be UTF-8 if sqlite3_open() or sqlite3_open_v2() is called and UTF-16 in the native byte order if sqlite3_open16() is used. > I see where I can specify UTF-16*. Is there support for UTF-32? No. Igor Tandetnik ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] UTF-8 BLOB
On Nov 6, 2007 8:55 AM, Joe Wilson <[EMAIL PROTECTED]> wrote: > It works in a Linux xterm. > There's probably some UTF or codepage issue with the Windows console. > Try using another command-line shell. > Thank you for your response Joe. It is a codepage issue. Using sqlite3.exe 3.5.2 from a Windows XP cmd.exe box, I tried SELECT blb FROM textblob WHERE blb IS NOT NULL; and got codepage 437 representations of the BLOB data in the blb field. Using the same query with tclsh and the tclsqlite3.dll 3.5.2 the output in the Windows XP cmd.exe box was a codepage 850 representation of the data. Also the sqlite3.exe 3.5.2 query SELECT dec, hex FROM textblob WHERE blb = 'À0'; worked when I substituted the codepage 437 glyphs for À0. I have yet to suceed with tclsh. Thanks again for the hint. I doubt that I would have remembered these codepage discrepancies if you hadn't reminded me. It has been several years since I last had this problem. Robert Wishlaw > --- Robert Wishlaw <[EMAIL PROTECTED]> wrote: > > > Using sqlite 3.5.2 on Windows XP, I have a textblob.csv file > > > > 192,C0,À0,À0 > > 193,C1,Á0,Á0 > > 254,FE,þ0,þ0 > > 255,FF,ÿ0,ÿ0 > > > > which I have imported into a new database > > > > sqlite3 textblob.db < textblob.sql > > > > via a file named textblob.sql > > > > .separator , > > create Table textblob(dec INTEGER, hex BLOB, txt TEXT, blb BLOB); > > .import 'textblob.csv' textblob > > > > When I run the query > > > > SELECT dec, hex FROM textblob WHERE blb = 'À0'; > > > > there is no result. Likewise > > > > SELECT dec, hex FROM textblob WHERE blb = "À0"; > > > > does not work. > > > > SELECT dec, hex FROM textblob WHERE hex = "C0"; > > > > returns > > > > 192|C0 > > > > Any idea how to get the BLOB data? Or is the problem that the BLOB > > data is not there because .import is filtering out invalid UTF-8? > > > > __ > Do You Yahoo!? > Tired of spam? Yahoo! Mail has the best spam protection around > http://mail.yahoo.com > > - > To unsubscribe, send email to [EMAIL PROTECTED] > - > > - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] UTF-8 BLOB
It works in a Linux xterm. There's probably some UTF or codepage issue with the Windows console. Try using another command-line shell. --- Robert Wishlaw <[EMAIL PROTECTED]> wrote: > Using sqlite 3.5.2 on Windows XP, I have a textblob.csv file > > 192,C0,À0,À0 > 193,C1,Á0,Á0 > 254,FE,þ0,þ0 > 255,FF,ÿ0,ÿ0 > > which I have imported into a new database > > sqlite3 textblob.db < textblob.sql > > via a file named textblob.sql > > .separator , > create Table textblob(dec INTEGER, hex BLOB, txt TEXT, blb BLOB); > .import 'textblob.csv' textblob > > When I run the query > > SELECT dec, hex FROM textblob WHERE blb = 'À0'; > > there is no result. Likewise > > SELECT dec, hex FROM textblob WHERE blb = "À0"; > > does not work. > > SELECT dec, hex FROM textblob WHERE hex = "C0"; > > returns > > 192|C0 > > Any idea how to get the BLOB data? Or is the problem that the BLOB > data is not there because .import is filtering out invalid UTF-8? __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] UTF-8 or ISO-8859-1
At 9:42 PM +0200 3/25/07, Ralph Müller wrote: to store (german) Text in a SQLite Database, is it better to use UTF-8 or is ISO-8859-1 more recommendable? It is recommended to use UTF-8 no matter what language text is in the database, since it is fully adaptable to any situation. Also, with SQlite 3, you don't get the second choice. -- Darren Duncan - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] UTF-8 or ISO-8859-1
Ralph Müller schrieb: Hi all, to store (german) Text in a SQLite Database, is it better to use UTF-8 or is ISO-8859-1 more recommendable? UTF-8. Frank - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] UTF-8 and ISO-8859-1 in the same library
On Fri, 30 Jul 2004, Bertrand Mansion wrote: >Hi, > >How difficult would it be to have both UTF-8 and ISO-8859-1 supported in >the same library (2.8.15) ? By looking quickly at the sources, it seems >to be just a matter of changing the functions likeFunc(), globFunc(), >lengthFunc() and substrFunc(). They could use a flag in order to call the >code required by the specified encoding. The problem is that the encoding isn't stored with the data, hence the library must be configured to assume one or the other. As the two are mutually incompatible, you're probably better off with UTF-8 encoding and converting strings yourself. I always compile SQLite v2 with UTF-8 encoding, as all the characters in ISO-8859-1 can be represented in UTF-8 (I believe. i18n is not my area of expertise.) > >In my case, this is needed as I can't figure out in advance which version >of the library the user wants to use. But my application lets the user >specify in which encodings the inputs are. Why not compile and bundle SQLite with your application? You have full control then. > >Is this the way to go or am I missing something ? > >Thanks in advance, > >Bertrand Mansion >Mamasam > -- /"\ \ /ASCII RIBBON CAMPAIGN - AGAINST HTML MAIL X - AGAINST MS ATTACHMENTS / \
