Re: [sqlite] sqlite3_column_text() returning partial results

2011-12-25 Thread Igor Tandetnik 
Matt Young  wrote:
> I used the strcpy_s function from Microsoft, the so called safe version
> that includes a char count. I used it under the Studio debugger.  Set a
> buffer of 200 chars to zero, set the char count to 20 in strcpy_s, and the
> debugger wrote in the top 180!!

As per documentation: "The debug versions of these functions first fill the 
buffer with 0xFD. To disable this behavior, use _CrtSetDebugFillThreshold." 
This is to make buffer overruns easier to spot under debugger. If you see a 
variable unexpectedly overwritten with 0xFD pattern, chances are high you are 
passing incorrect buffer length to strcpy_s or similar, and the buffer happens 
to be located in memory right before that varible.
-- 
Igor Tandetnik

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] sqlite3_column_text() returning partial results

2011-12-25 Thread Matt Young 
I used the strcpy_s function from Microsoft, the so called safe version
that includes a char count. I used it under the Studio debugger.  Set a
buffer of 200 chars to zero, set the char count to 20 in strcpy_s, and the
debugger wrote in the top 180!!

I freaked, didn't see that as safe at all!

On Sat, Dec 24, 2011 at 4:28 AM, Tony McC  wrote:

> On Wed, 21 Dec 2011 21:33:02 +
> "Jacob A. Camp"  wrote:
>
> > There it is, when I printed this value out to a txt file everything
> > was there.
> >
> > Apparently there is some sort of character limit when displaying
> > strings in debug mode in Visual Studio 2010, news to me...
> >
> > Thanks everyone!
>
> That's what the 2010 in the name of the product means: displayed strings
> are limited to that many characters (Studio is an acronym meaning Stop
> Displaying Under Debugger If Over [2010]). You were lucky to get a few
> more, but that is undocumented behaviour.  If you upgrade next year
> though you should be able to display a couple more characters.
>
> Tony
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] sqlite3_column_text() returning partial results

2011-12-24 Thread Tony McC 
On Wed, 21 Dec 2011 21:33:02 +
"Jacob A. Camp"  wrote:

> There it is, when I printed this value out to a txt file everything
> was there. 
> 
> Apparently there is some sort of character limit when displaying
> strings in debug mode in Visual Studio 2010, news to me...
> 
> Thanks everyone!

That's what the 2010 in the name of the product means: displayed strings
are limited to that many characters (Studio is an acronym meaning Stop
Displaying Under Debugger If Over [2010]). You were lucky to get a few
more, but that is undocumented behaviour.  If you upgrade next year
though you should be able to display a couple more characters.

Tony
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] sqlite3_column_text() returning partial results

2011-12-21 Thread Jacob A. Camp 
There it is, when I printed this value out to a txt file everything was there. 

Apparently there is some sort of character limit when displaying strings in 
debug mode in Visual Studio 2010, news to me...

Thanks everyone!

-Original Message-
From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] 
On Behalf Of Pavel Ivanov
Sent: Wednesday, December 21, 2011 2:33 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] sqlite3_column_text() returning partial results

Now, this is a very good test case and explanation what happens, except...

How did you check value of CString testing after iterating though the entire 
char array? Did you look at it in your debugger? Maybe your debugger is not 
capable of showing CString contents longer than 2048 symbols? The value of 
string testing you gave with Windows-style line ends gives me 2040 bytes, 
indentation of the next line is 8 bytes, so if total limit is 2048 bytes then 
you won't see any contents of the next line...

Can you output contents of CString testing using printf() and see if it shows 
the same 2040 bytes?


Pavel


On Wed, Dec 21, 2011 at 1:34 PM, Jacob A. Camp <jacob.c...@mastercam.com> wrote:
> So delving further into this has gotten me nothing else; this is the best I 
> could do to simplify this to understandable steps.
>
> (I'm new to this service so no idea if attachments are allows so sorry 
> about the length in advance)
>
> I have this code:
>
>                sqlite3_stmt* state = NULL;
>                CString sql = _T("SELECT RelationshipHierarchyXML FROM 
> TlAssembly WHERE id=?");
>                //Query the database
>                RawSQLTlID (sql, id, );
>                bytes = sqlite3_column_bytes(state, 0); //This line 
> returns 8955
>                //Try getting data as blob
>                const void * temp = sqlite3_column_blob(state, 0);
>                //Set the size for simplification
>                char chararray[9000];
>                //Copy value into the char array
>                memcpy( chararray , temp , bytes);
>                CString testing;
>                //Iterate through all elements
>                for (int x = 0; x < 8999; x++)
>                {
>                int temp = chararray[x];
>                char temp2 = chararray[x];
>
>                //if the char is null or not supportted don't append it 
> to CString
>                if (temp < 1 || temp > 255)
>                {
>                //This gets hit a few times with negative int values
>                temp = temp;
>                }
>                else
>                {
>                //Append the char to the XML string
>                testing.AppendChar(temp2);
>                }
>                }
>
>                return success;
>
> At this point the CString testing contains on a portion of the characters of 
> the XML field which makes it unusable.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

**
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

This footnote also confirms that this email message has been swept by
MIMEsweeper for the presence of computer viruses.

www.mastercam.com
**
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] sqlite3_column_text() returning partial results

2011-12-21 Thread Pavel Ivanov 
Now, this is a very good test case and explanation what happens, except...

How did you check value of CString testing after iterating though the
entire char array? Did you look at it in your debugger? Maybe your
debugger is not capable of showing CString contents longer than 2048
symbols? The value of string testing you gave with Windows-style line
ends gives me 2040 bytes, indentation of the next line is 8 bytes, so
if total limit is 2048 bytes then you won't see any contents of the
next line...

Can you output contents of CString testing using printf() and see if
it shows the same 2040 bytes?


Pavel


On Wed, Dec 21, 2011 at 1:34 PM, Jacob A. Camp  wrote:
> So delving further into this has gotten me nothing else; this is the best I 
> could do to simplify this to understandable steps.
>
> (I'm new to this service so no idea if attachments are allows so sorry about 
> the length in advance)
>
> I have this code:
>
>                sqlite3_stmt* state = NULL;
>                CString sql = _T("SELECT RelationshipHierarchyXML FROM 
> TlAssembly WHERE id=?");
>                //Query the database
>                RawSQLTlID (sql, id, );
>                bytes = sqlite3_column_bytes(state, 0); //This line returns 
> 8955
>                //Try getting data as blob
>                const void * temp = sqlite3_column_blob(state, 0);
>                //Set the size for simplification
>                char chararray[9000];
>                //Copy value into the char array
>                memcpy( chararray , temp , bytes);
>                CString testing;
>                //Iterate through all elements
>                for (int x = 0; x < 8999; x++)
>                {
>                int temp = chararray[x];
>                char temp2 = chararray[x];
>
>                //if the char is null or not supportted don't append it to 
> CString
>                if (temp < 1 || temp > 255)
>                {
>                //This gets hit a few times with negative int values
>                temp = temp;
>                }
>                else
>                {
>                //Append the char to the XML string
>                testing.AppendChar(temp2);
>                }
>                }
>
>                return success;
>
> At this point the CString testing contains on a portion of the characters of 
> the XML field which makes it unusable.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] sqlite3_column_text() returning partial results

2011-12-21 Thread Jacob A. Camp 
  1
0
0
0
0
1
0
0
0
0
1
  
  

  true
  
I
1
0
0
0
0
1
0
0
0
0
1
0
0
2.5
0
1
  
  Target
  DefaultTarget
  
I
1
0
0
0
0
1
0
0
0
0
1
0
0
0
0
1
  
  


  true
  
I
1
0
0
0
0
1
0
0
0
0
1
0
0
0
0
1
  
  LinearAxis
  
  
I
1
0
0
0
0
1
0
0
0
0
1
0
0
0
0
1
  
  

  true
  
I
1
0
0
0
0
1
0
0
0
0
1
0
0
0
0
1
  
  Real Holder
  
  
I
1
0
0
0
0
1
0
0
0
0
1
0
0
0
0
1
  
  

  true
  
I
1
0
0
0
0
1
0
0
0
0
1
0
0
6.12099987
0
1
  
  Target
  DefaultTarget
  
I
1
0
0
0
0
1
0
0
0
0
1
0
0
0
0
1
  
  

  
  #6699cc
  1
  Holder
  0
  d22d6e4c-be03-471e-bcac-0e5d12852fb2

  
  0
  1
  0
  0
  2.5
  


  
  0.75
  1.3118700546473194
  Y

  
  #6699cc
  1
  Tool
  1.3118700546473194
  ----

  
  
  
  false
  VerticalLathe
  
  
  
  
",

-----Original Message-
From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] 
On Behalf Of Pavel Ivanov
Sent: Wednesday, December 21, 2011 11:20 AM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] sqlite3_column_text() returning partial results

On Wed, Dec 21, 2011 at 10:41 AM, Jacob A. Camp <jacob.c...@mastercam.com> 
wrote:
> I have a line of code that executes that line:
>
> const unsigned char * temp2 = sqlite3_column_text(state, 0);
>
> This queries the database and after the call is complete I pass this value to 
> another function. This function then fails because temp2 points to a location 
> that contains the incomplete text. I then can view this text using my 
> debugger and I can see that the value has been clipped and the XML is invalid.

Let me ask Igor's questions once again. How does your function understand that 
text is incomplete? Your debugger can show you incomplete string because temp2 
is not a zero-terminated string, it can contain zeros in the middle.

> Trying to access memory locations outside whatever is allocated by the return 
> of sqlite3_column_text sounds like it would result in undefined behavior?

How do you know how much memory sqlite3_column_text allocated for the return?

> sqlite3_column_bytes returns 8960 if that's helpful.

Hint: the above number means that sqlite3_column_text allocated exactly 8960 
bytes f

Re: [sqlite] sqlite3_column_text() returning partial results

2011-12-21 Thread Pavel Ivanov 
On Wed, Dec 21, 2011 at 10:41 AM, Jacob A. Camp
 wrote:
> I have a line of code that executes that line:
>
> const unsigned char * temp2 = sqlite3_column_text(state, 0);
>
> This queries the database and after the call is complete I pass this value to 
> another function. This function then fails because temp2 points to a location 
> that contains the incomplete text. I then can view this text using my 
> debugger and I can see that the value has been clipped and the XML is invalid.

Let me ask Igor's questions once again. How does your function
understand that text is incomplete? Your debugger can show you
incomplete string because temp2 is not a zero-terminated string, it
can contain zeros in the middle.

> Trying to access memory locations outside whatever is allocated by the return 
> of sqlite3_column_text sounds like it would result in undefined behavior?

How do you know how much memory sqlite3_column_text allocated for the return?

> sqlite3_column_bytes returns 8960 if that's helpful.

Hint: the above number means that sqlite3_column_text allocated
exactly 8960 bytes for the text it returned. So whatever you use to
define that returned text is incomplete gives you incorrect
information.


Pavel
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] sqlite3_column_text() returning partial results

2011-12-21 Thread Black, Michael (IS) 
What does strlen() tell you on the XML before you put it in the table?



Can you boil all this down to one record in a table, and some code so we can 
all see what's going on?

Do a .dump of the table and post the code.



See if you can reproduce it in a simple example.



Michael D. Black

Senior Scientist

Advanced Analytics Directorate

Advanced GEOINT Solutions Operating Unit

Northrop Grumman Information Systems


From: sqlite-users-boun...@sqlite.org [sqlite-users-boun...@sqlite.org] on 
behalf of Jacob A. Camp [jacob.c...@mastercam.com]
Sent: Wednesday, December 21, 2011 9:41 AM
To: 'General Discussion of SQLite Database'
Subject: EXT :Re: [sqlite] sqlite3_column_text() returning partial results

I have a line of code that executes that line:

const unsigned char * temp2 = sqlite3_column_text(state, 0);

This queries the database and after the call is complete I pass this value to 
another function. This function then fails because temp2 points to a location 
that contains the incomplete text. I then can view this text using my debugger 
and I can see that the value has been clipped and the XML is invalid.

Trying to access memory locations outside whatever is allocated by the return 
of sqlite3_column_text sounds like it would result in undefined behavior?

sqlite3_column_bytes returns 8960 if that's helpful.

-Original Message-
From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] 
On Behalf Of Igor Tandetnik
Sent: Wednesday, December 21, 2011 10:29 AM
To: sqlite-users@sqlite.org
Subject: Re: [sqlite] sqlite3_column_text() returning partial results

Jake <jacob.c...@mastercam.com> wrote:
> For the previous post asking what occurs after 2030, there's nothing

What do you mean, nothing? What exactly happens when you try to access p[2030]?

> because the
> returned value is only allocated for those characters.

What makes you believe this?
--
Igor Tandetnik

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

**
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

This footnote also confirms that this email message has been swept by
MIMEsweeper for the presence of computer viruses.

www.mastercam.com<http://www.mastercam.com/>
**
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] sqlite3_column_text() returning partial results

2011-12-21 Thread Jacob A. Camp 
I have a line of code that executes that line:

const unsigned char * temp2 = sqlite3_column_text(state, 0);

This queries the database and after the call is complete I pass this value to 
another function. This function then fails because temp2 points to a location 
that contains the incomplete text. I then can view this text using my debugger 
and I can see that the value has been clipped and the XML is invalid. 

Trying to access memory locations outside whatever is allocated by the return 
of sqlite3_column_text sounds like it would result in undefined behavior?

sqlite3_column_bytes returns 8960 if that's helpful.

-Original Message-
From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] 
On Behalf Of Igor Tandetnik
Sent: Wednesday, December 21, 2011 10:29 AM
To: sqlite-users@sqlite.org
Subject: Re: [sqlite] sqlite3_column_text() returning partial results

Jake <jacob.c...@mastercam.com> wrote:
> For the previous post asking what occurs after 2030, there's nothing

What do you mean, nothing? What exactly happens when you try to access p[2030]?

> because the
> returned value is only allocated for those characters.

What makes you believe this?
-- 
Igor Tandetnik

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

**
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

This footnote also confirms that this email message has been swept by
MIMEsweeper for the presence of computer viruses.

www.mastercam.com
**
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] sqlite3_column_text() returning partial results

2011-12-21 Thread Igor Tandetnik 
Jake  wrote:
> For the previous post asking what occurs after 2030, there's nothing

What do you mean, nothing? What exactly happens when you try to access p[2030]?

> because the
> returned value is only allocated for those characters.

What makes you believe this?
-- 
Igor Tandetnik

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] sqlite3_column_text() returning partial results

2011-12-21 Thread Simon Slavin 

On 21 Dec 2011, at 1:31pm, nobre wrote:

> Is there any chance you are storing a \0 char inside the xml ?

Or that you are mixing 8-bit and 16-bit Unicode in such a way that one of your 
routines thinks that it has read a 0x00 termination character ?

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] sqlite3_column_text() returning partial results

2011-12-21 Thread nobre 

Is there any chance you are storing a \0 char inside the xml ? 

Jacob A. Camp wrote:
> 
> Hello,
> 
> I've been looking into an issue that a few of our programmers have looked
> at as well and it left us all wondering. Basically, our database has a
> VARCHAR column that has an XML file written to it when the object is
> manipulated. I can use a tool to view the database file and I can ensure
> that the entire field is filled out correctly and the XML is correctly
> formed.
> 
> The field in the database contains 8955 characters and when I execute the
> sqlite3_column_text() on that specific column to access the data, the
> const unsigned char* that's returned only contains 2030 characters and the
> XML file that I'm trying to reconstruct from it then becomes unusable. I
> even attempted digging into the SQLite class and it seems that this value
> is obtained from the official function calls.
> 
> In sqlite3.c:
> 
> static Mem *columnMem(sqlite3_stmt *pStmt, int i){
>   Vdbe *pVm;
>   int vals;
>   Mem *pOut;
> 
>   pVm = (Vdbe *)pStmt;
>   if( pVm && pVm->pResultSet!=0 && inResColumn && i>=0 ){
> sqlite3_mutex_enter(pVm->db->mutex);
> vals = sqlite3_data_count(pStmt);
> pOut = >pResultSet[i];
> 
> After the last line has been executed pOut contains members z and zMalloc
> which both contain the same memory location that points to the char* that
> contains the first 2030 characters and none of the rest that are stored in
> the VARCHAR field.
> 
> Is there some limit to the amount of data that can be returned by this
> function? I read over most of the documentation and didn't see it
> mentioning any sort of restriction.
> 
> Thanks in advance,
> --Jake
> 
> **
> This email and any files transmitted with it are confidential and
> intended solely for the use of the individual or entity to whom they
> are addressed. If you have received this email in error please notify
> the system manager.
> 
> This footnote also confirms that this email message has been swept by
> MIMEsweeper for the presence of computer viruses.
> 
> www.mastercam.com
> **
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
> 
> 

-- 
View this message in context: 
http://old.nabble.com/sqlite3_column_text%28%29-returning-partial-results-tp33016613p33016758.html
Sent from the SQLite mailing list archive at Nabble.com.

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] sqlite3_column_text() returning partial results

2011-12-21 Thread Igor Tandetnik 
Jacob A. Camp  wrote:
> The field in the database contains 8955 characters and when I execute the 
> sqlite3_column_text() on that specific column to access
> the data, the const unsigned char* that's returned only contains 2030 
> characters

What do you mean by that? sqlite3_column_text doesn't provide any length 
delimiter. What's in p[2030], p[2031] and so on (where p is a pointer obtained 
from sqlite3_column_text)?

Are you examining the string immediately after calling sqlite3_column_text? 
Some sqlite3_* calls invalidate the pointer (by reusing or freeing the memory 
behind it).

What does sqlite3_column_bytes return for this column?
-- 
Igor Tandetnik

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users