Pavel:
Though I am a newbie to SQLite I am not a newbie to programming. Please
keep that in mind in any future interaction.
I did indeed look at the data through the SQLite command utility before I
offered my answer below, and it supported my conclusion. And no, I am not
going to provide you with output. If my testimony that it worked is not enough
for you, then...well I'm sorry about that.
The WideString object InDesign uses is its own WideString class that is part
of the InDesign SDK. The headers for both WideString and PMString objects (the
string object that InDesign uses primarily for UI purposes), declare that a
WideString may be converted to a PMString simply by using the PMString
constructor and the WideString as its argument. So I simply did this:
WideString SomeWString("With some widestring data in it");
PMString SomePMString(SomeWideString);
std::string SomeString = SomePMString.GetPlatformString().
PMString::GetPlatformString() provides the user with a std::basic_string on
US-standard Mac and Windows boxes. I suspect that on computers that use Kanji
or other non-western alphabets, it provides strings appropriate to their
language. That's why it's called "GetPlatformString."
Oh and by the way, I was using the STL basic_string<char> string object. I
have no idea what other kind of basic_string you thought I might be using.
The PMString does have enough flexibility to use 16-bit chars, but it also
has enough intelligence to use 8-bit chars if the language warrants it (and a
few other rules that are not clear but are obviously in play to someone who has
had to work with PMString for the last eight years or so). Besides, I only
used PMString as a stepping stone to get from WideString to basic_string,
which, by the way, I looked at in the debugger and found it to be standard
ASCII characters--no spaces or unexpected characters included except at the
expected places.
I should think that the fact that I was able to copy the string using the
old-style strcpy(), and found its true length with strlen(), ending up with the
expected data in the tables would have finally been proof enough to you that I
knew what I was talking about, since strcpy is a simple byte-by-byte copy
function.
The locked-in-concrete size of the char buf that you point out is a good
point. However, the code was proof of concept code. I was simply trying to
figure out how to make something work. I wasn't interested in writing clean,
elegant code with all the 'i's dotted and the 't's crossed. Programmers often
do that when they're first wading into unknown waters.
When you talk about pointers, there are all sorts of ways in which a pointer
can be different and yet be functionally the same. Consider the following:
Char *p;
Char p[100];
Const char *p;
Const char p[100];
Void *p; // cast to something like this: (char *) it would work just fine
as a null-terminated C-style string.
One could even do something like this, though why he/she might want to is a
mystery (but I have seen stranger and more wondrous code):
Smallint q[100];
Int a = strlen((char *) q);
(Please note that Outlook is capitalizing the first words of the line--I
recognize that they all should be lower case-so please don't be petty and point
it out.)
In C++ if a function calls for a *p and you give it a p[], the compiler will
complain. The compiler did not complain when I used std::basic_string.c_str(),
which is why this is all so puzzling. Yet it is obvious to me, even if it is
not to you, that the data was fine (not UTF16). sqlite3_bind_text simply did
not like the std::basic_string.c_str() pointer returned. But a standard,
C-style null-terminated string worked fine.
If the SQLite3 function expects the first kind of pointer and goes crazy
with anything else, then that could have been my problem.
I hope I have answered all of your questions. As far as I am concerned, my
problem is solved so unless you have some new insight that will greatly speed
my program or enhance efficiency, you do not have to respond to this email.
R,
John
> -----Original Message-----
> From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org]
> On Behalf Of Pavel Ivanov
> Sent: Saturday, December 18, 2010 9:28 PM
> To: General Discussion of SQLite Database
> Subject: Re: [sqlite] Troubleshooting...
>
> > Okay. I worked out a solution, but I am still unsure of why it fixes the
> > problem.
>
> Does it really fix your problem? Can you provide output from command
> line utility before the fix and after? I don't see how it can fix
> anything.
>
> > Nevertheless your point is well-taken that the data could be something
> > other
> than UTF8, especially since InDesign uses a WideString class internally.
>
> Which means that you most probably deal with UTF-16 and maybe need to
> use sqlite3_bind_text16 instead of sqlite3_bind_text.
>
> > I'm pretty sure I have taken the steps that need to be taken to transform
> > the
> WideString objects to basic_string objects, but that is an assumption that
> needs to
> be tested, and I am working on a way to do that right now.
>
> Could you show us how did you transform WideString objects into string ones?
>
> > Supposedly, basic_string deals with nothing other than UTF8 strings. That
> > has
> been my assumption until now. I will let you know what I figure out. Soon.
>
> You are wrong here. First of all don't talk about basic_string when
> you talk about string which is basic_string<char> - that makes a lot
> of difference. So string deals with nothing other than sequence of
> bytes. It doesn't know anything about encodings and doesn't even know
> if the data it has is text or some binary stuff. It knows only a
> sequence of bytes and its length. So if you put into string something
> in UTF-16 encoding it will work with that without problems, although
> SQLite won't understand it if it expects UTF-8.
>
> > My only guess is that basic_string::c_str() doesn't really provide a
> > pointer to a
> null-terminated c-style string, but a facsimile of one that SQLite doesn't
> like.
>
> c_str() provides pointer to the data that string has with additional
> null byte added at the end. That's it. Whether it is a null-terminated
> c-style string depends on what data you put into it. If that data has
> null bytes in it then SQLite will see only part of your string, though
> I'm not sure that it was the actual problem you have seen.
>
> > char *surbuf[100];
> > memset(surbuf, 0, 100);
> > strcpy(surbuf, CurrentName -> second.GetSurName().c_str());
>
> Don't do this in production application as it can overflow your surbuf
> (if GetSurName() returns something that is longer than 100 bytes).
> Apart from that this statement and call to strlen() further do nothing
> different than SQLite does by itself when you provide to it simple
> result of c_str() and let it determine length of the string. So this
> definitely can't fix any problem. I guess you somehow changed result
> of GetSurName() on the way here.
>
>
> Pavel
>
> On Fri, Dec 17, 2010 at 6:18 PM, john darnell
> <john.darn...@walsworth.com> wrote:
> > Okay. I worked out a solution, but I am still unsure of why it fixes the
> > problem.
> Or rather, I know why it works, but it still doesn't explain why the
> basic_string::c_str() call did not work. My only guess is that
> basic_string::c_str()
> doesn't really provide a pointer to a null-terminated c-style string, but a
> facsimile of
> one that SQLite doesn't like.
> >
> >
> > Here's that statement from my code again with the mods included that make
> > the
> code work:
> >
> > char *surbuf[100];
> > memset(surbuf, 0, 100);
> > strcpy(surbuf, CurrentName -> second.GetSurName().c_str());
> > idx = -1;
> > idx = sqlite3_bind_parameter_index(ResultStmt, ":sur");
> > sqlite3_bind_text(ResultStmt, idx, surbuf, strlen(surbuf),
> > SQLITE_STATIC);
> >
> >
> > BTW, I was using the .output command in the SQLITE3 shell to attempt piping
> my output to a file. The command looked like this:
> >
> > .output c:\sqlite.txt
> >
> > Select * from Names;
> >
> > When I went to the root of C: I did not find the file. So I removed the
> > path from
> the filename and later found it in the SQLLite directory from which I was
> running
> the shell.
> >
> > R,
> > John
> >
> >> -----Original Message-----
> >> From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-
> boun...@sqlite.org]
> >> On Behalf Of Pavel Ivanov
> >> Sent: Friday, December 17, 2010 4:44 PM
> >> To: General Discussion of SQLite Database
> >> Subject: Re: [sqlite] Troubleshooting...
> >>
> >> > So you can see that when I add the hard-coded data, everything looks
> >> > fine in
> the
> >> results of the select statement, which leads me to believe that the
> >> problem is
> not
> >> confusion between UTF8 and UTF16 output.
> >>
> >> Your hardcoded data has only characters from ASCII set which don't
> >> differ from the same in UTF-8. Besides SQLite API and command line
> >> utility don't check your strings to be valid UTF-8. SQLite API puts
> >> into database whatever you give to it, command line utility throws to
> >> stdout whatever it finds in the database (so my guess could be wrong
> >> and your input strings could be not in UTF-8 but in some other
> >> encoding). It's totally developer's responsibility to make sure that
> >> encoding put into database is the same as is expected when it's
> >> retrieved from database.
> >>
> >> > I could not figure out how to pipe my info to a file
> >>
> >> One of possible solutions is to use .output command in the sqlite3
> >> utility (you can use .help command to see everything that is available
> >> to you there) or to provide your sql statement as last argument when
> >> you start sqlite3 utility and use standard redirection like this:
> >>
> >> sqlite3 database.db "select * from mytable" >output.txt
> >>
> >
> > _______________________________________________
> > sqlite-users mailing list
> > sqlite-users@sqlite.org
> > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
> >
> _______________________________________________
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
_______________________________________________
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
