On Tue, Oct 7, 2014 at 2:20 PM, jose isaias cabrera <jic...@cinops.xerox.com > wrote:
> > "J Decker" wrote... > > > So, I guess it is technically not allowed to encode 11 bit unicode >> characters as 16. >> the greek characters are 0x3XX which is 10 bits... I checked what >> WideCharToMultiByte was doing and found it was using 11 bit encodings... >> fixed my encoder to use an appropriate size for what's required, added 11 >> bit decoding, and now in and out works for that and some chinese >> characters >> which are more than 11 bits. >> >> The 'unrecognized token' is 0xE0 ? ... although a thing could be 12 bits >> exactly... so is it checking ( char[0] == 0xe0 ) && ( ( char[1] & 0xE0 ) >> == >> 0x80 )? >> >> as a side note.. using visual studio to mouse over the resulting char * >> string with 11 bit encodings it shows bad characters, if encoded as (valid >> but illegal) 16 bit it browses correctly. >> > > J, > > My suggestion is for you to read about ANSI, ASCII, UTF7, UTF8, UTF16 and > UTF32 and understand the ins and outs of the various encoding. You may need > to create your own wrapper to get things to work correctly. > Right; I did, and have, but missed the part 'must be encoded in least bits' (and I'm not sure it is there, and visual studio sees it as a valid thing to do; to use a encoding larger than the number of bits required) and unicode only uses 20 bits max so the extended 5 and 6 byte utf-8 encodings never get used Have a custom wrapper for systems that are not windows; and now it's more robust. Still think it's something of a bug, but has been worked around so for me it won't be an issue again. > > What happens if you create a text file using notepad and make sure that > you save it as UTF8 and then read that file with the content that you want? > Then write it to SQLite and get it back and write it back to another file? > Does that work? > _______________________________________________ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
