[sqlite] Issue with sqlite3_result_error inside aggregate function
I am seeing a process go down. The issue is happening if sqlite3_result_error is called from within the implementation of an aggregate function during a step. The stack trace looks like this (against 3.2.7 code): ==2378== Conditional jump or move depends on uninitialised value(s) ==2378==at 0x1BE9E175: sqlite3VdbeMemRelease (vdbemem.c:220) ==2378==by 0x1BE9E556: sqlite3VdbeMemSetStr (vdbemem.c:416) ==2378==by 0x1BE9AC17: sqlite3_result_error (vdbeapi.c:96) ==2378==by 0x1BE8E47C: cbdispatch_step (apsw.c:1311) ==2378==by 0x1BEC9351: sqlite3VdbeExec (vdbe.c:4250) ==2378==by 0x1BE9AF49: sqlite3_step (vdbeapi.c:217) ==2378==by 0x1BE90485: Cursor_step (apsw.c:2252) ==2378==by 0x1BE90A70: Cursor_execute (apsw.c:2445) ==2378==by 0x1B96DB3A: PyCFunction_Call (in /usr/lib/libpython2.3.so.1.0) Roger
Re: [sqlite] Issue with sqlite3_result_error inside aggregate function
On Mon, 2005-09-26 at 23:16 -0700, Roger Binns wrote: I am seeing a process go down. The issue is happening if sqlite3_result_error is called from within the implementation of an aggregate function during a step. The stack trace looks like this (against 3.2.7 code): You should not call sqlite3_result_* from within the step function of an aggregate. Those routines may be called from within the finalizer function only. -- D. Richard Hipp [EMAIL PROTECTED]
Re: [sqlite] Assertion failure in btree.c, line 1166
Richard, Thanks for looking into this. It's actually compiled into PySQLite so I can't determine it directly. I'll ask on the PySQLite mailing list. Hugh *Subject:* Re: [sqlite] Assertion failure in btree.c, line 1166 *From:* D. Richard Hipp [EMAIL PROTECTED] *To:* [EMAIL PROTECTED] *Date:* Tue, 27 Sep 2005 06:37:44 -0400 On Mon, 2005-09-26 at 12:03 +0100, Hugh Gibson wrote: Hi, Using PySQLite 2.0.4 I get a crash with the following sequence, starting from no database. If the page_size pragma is removed it works OK. Reproducible under Windows 2K and XP. Running Python 2.4.1. You have not indicated what version of SQLite you are using. I am unable to reproduce the problem. I run your script and it works.
[sqlite] How to cross compile sqlite-3.2.7 for arm9 s3c2410?
I have downloaded sqlite-3.2.7.tar.gz from Internet, but I do not know how to compile it on s3c2410(linux 2.6, arm9, samsung)? There is no useful tips in README or INSTALL, pls help me. Thx. -- 我现在使用Sogou.com的2G邮箱了，你也来试试吧! http://mail.sogou.com/recommend/sogoumail_invite_reg1.jsp?from=sogouinvitations_EMAIL=cupidsd%40sogou.comusername=sqlite-usersFullName=sqlite-usersEmail=sqlite-users%40sqlite.orgverify=fcd64dc521a05fbe8f224ceef58ce291
Re: [sqlite] Enhancements for SQLite
On 9/27/05, Jay Sprenkle [EMAIL PROTECTED] wrote: I'm developing an open source project (a Data Access Grid) that uses internally the SQLite library as the storage method and I'm interested in your opinion about 4 questions: 1) SQLite can't deal with raw devices. Should be hard to patch the source to deal with raw devices? You'll have to write your own file system code (with locking). Seriously though, there's no reason for this. It's not going to help on performance unless you also add all sorts of multi-spindle management code. Having worked a lot with Oracle on raw-devices, it's only beneficial when you can throw lots of different spindles at it. Chris -- | Christopher Petrilli | [EMAIL PROTECTED]
Re: [sqlite] Enhancements for SQLite
On Tue, 2005-09-27 at 13:40 +0200, Joxean Koret wrote: Why not use mmap, munmap, mprotect, mlock, etc... system calls? You ever tried to mmap a 10GiB database file into the memory of processor with a 4GiB address space? -- D. Richard Hipp [EMAIL PROTECTED]
Re: [sqlite] Enhancements for SQLite
On 9/27/05, Christopher Petrilli [EMAIL PROTECTED] wrote: On 9/27/05, Jay Sprenkle [EMAIL PROTECTED] wrote: I'm developing an open source project (a Data Access Grid) that uses internally the SQLite library as the storage method and I'm interested in your opinion about 4 questions: 1) SQLite can't deal with raw devices. Should be hard to patch the source to deal with raw devices? You'll have to write your own file system code (with locking). Seriously though, there's no reason for this. It's not going to help on performance unless you also add all sorts of multi-spindle management code. Having worked a lot with Oracle on raw-devices, it's only beneficial when you can throw lots of different spindles at it. It might be good if you're implementing on many operating systems that don't all do file system locking the same way or even at all...
RE: [sqlite] any suggestion for the database file extension?mine is xxx.drh
I use .db3 At least one tool, Sqlite3Explorer, assumes that extension if I am not mistaken. Shawn M. Downey MPR Associates 10 Maxwell Drive, Suite 204 Clifton Park, NY 12065 518-371-3983 x3 (work) 860-508-5015 (cell) -Original Message- From: Huanghongdong [mailto:[EMAIL PROTECTED] Sent: Tuesday, September 27, 2005 10:05 AM To: sqlite-users Subject: [sqlite] any suggestion for the database file extension?mine is xxx.drh sqlite-users，您好！ 致 礼！ Huanghongdong [EMAIL PROTECTED] 2005-09-27
Re: Fw: [sqlite] bestIndex with Float values
On Tue, 2005-09-27 at 17:14 +0530, Sankara Narayanan wrote: A gentle reminder requesting for clarification. If our processor does not support floating points how do I modify the bestIndex function so that the functionality is not affected. (1) Get a compile (such as gcc) that will emulate floating point. (2) Modify the code to use integers rather than doubles to score indices in the optimizer. -- D. Richard Hipp [EMAIL PROTECTED]
Re: [sqlite] any suggestion for the database file extension?mine is xxx.drh
Hi, several tools use .db3 (for sqlite3 - Format) Martin Huanghongdong schrieb: sqlite-users，您好！ 致 礼！ Huanghongdong [EMAIL PROTECTED] 2005-09-27
Re: [sqlite] compression
If you're on Linux read about the Reiser 4 file system. They found they could compress the entire file system on the fly and achieve higher performance as well. Most CPU's can compress and move data faster because they make up the difference on the slow I/O channels to hard disks. Might be a much easier solution On 9/24/05, Martin Pfeifle [EMAIL PROTECTED] wrote: Hello, does anybody know whether it is possible to compress sqlite data on the page level. If I compress the sqlite database file with zlib I get very high compression rates due to the character of the stored data. I think this problem is related to the problem of using encrypted databases. Perheps it is possible just to exchange the encryption function call by a zlib compression call. Integrating such a call simply into the read and write functions in the file os_win.c does not work. Can anybody help me, or give me a hint? Ciao Martin ___ Was denken Sie über E-Mail? Wir hören auf Ihre Meinung: http://surveylink.yahoo.com/wix/p0379378.aspx -- --- The Castles of Dereth Calendar: a tour of the art and architecture of Asheron's Call http://www.lulu.com/content/77264
Re: [sqlite] How to cross compile sqlite-3.2.7 for arm9 s3c2410?
I have downloaded sqlite-3.2.7.tar.gz from Internet, but I do not know how
to compile it on s3c2410(linux 2.6, arm9, samsung)?
There's some info in the wiki on it, but basically the steps are:
1. change the configure script (see attached patch for 3.2.6)
2. configure with something like this (in my case an arm sa1110):
cd $PATH/sqlite-3.2.6
CC=arm-sa1100-linux-gnu-gcc CXX=arm-sa1100-linux-gnu-g++ \
CFLAGS=-I$TOOLCHAIN_BASE/include BUILD_CC=gcc \
./configure --host=i686-pc-linux --target=arm-sa1100-linux \
--disable-shared --disable-tcl --prefix=$PATH_INSTALL
3. change the Makefile a bit, setting 'BCC' to your regular compiler and (if
you want) add the static build flag, for example doing something like this:
mv Makefile Makefile.sed
cat Makefile.sed | sed 's/^BCC.*/BCC = gcc/' | sed 's/^LTLINK.*/
-all-static/' Makefile
rm Makefile.sed
4. make make install
Mind you, depending on which core you have and whether you want floating
point, you might find the other patch attached usefull as well. It is from an
earlier posting of myself:
In order to wrap this up: apparently there's a feature / bug (choose one) in
any ARM core earlier than v5 due to which a float will be stored in big
endian quad order. The processor in this particular case is an SA1110, which
is default little endian while having a v4 core. (and thus is 'swapping'
the quads).
--
Best,
Frank.
--- configure_orig 2005-09-19 10:46:30.0 +0200
+++ configure 2005-09-19 10:46:34.0 +0200
@@ -19209,11 +19209,11 @@
ac_link='$CC -o conftest$ac_exeext $CFLAGS $CPPFLAGS $LDFLAGS conftest.$ac_ext $LIBS 5'
ac_compiler_gnu=$ac_cv_c_compiler_gnu
- if test $cross_compiling = yes; then
-{ { echo $as_me:$LINENO: error: unable to find a compiler for building build tools 5
-echo $as_me: error: unable to find a compiler for building build tools 2;}
- { (exit 1); exit 1; }; }
- fi
+# if test $cross_compiling = yes; then
+#{ { echo $as_me:$LINENO: error: unable to find a compiler for building build tools 5
+#echo $as_me: error: unable to find a compiler for building build tools 2;}
+# { (exit 1); exit 1; }; }
+# fi
BUILD_CC=$CC
default_build_cflags=$CFLAGS
else
@@ -20165,11 +20165,11 @@
echo $ECHO_N checking for $dir/include/readline.h... $ECHO_C 6
if eval test \\${$as_ac_File+set}\ = set; then
echo $ECHO_N (cached) $ECHO_C 6
-else
- test $cross_compiling = yes
- { { echo $as_me:$LINENO: error: cannot check for file existence when cross compiling 5
-echo $as_me: error: cannot check for file existence when cross compiling 2;}
- { (exit 1); exit 1; }; }
+#else
+# test $cross_compiling = yes
+# { { echo $as_me:$LINENO: error: cannot check for file existence when cross compiling 5
+#echo $as_me: error: cannot check for file existence when cross compiling 2;}
+# { (exit 1); exit 1; }; }
if test -r $dir/include/readline.h; then
eval $as_ac_File=yes
else
@@ -20191,11 +20191,11 @@
echo $ECHO_N checking for $dir/include/readline/readline.h... $ECHO_C 6
if eval test \\${$as_ac_File+set}\ = set; then
echo $ECHO_N (cached) $ECHO_C 6
-else
- test $cross_compiling = yes
- { { echo $as_me:$LINENO: error: cannot check for file existence when cross compiling 5
-echo $as_me: error: cannot check for file existence when cross compiling 2;}
- { (exit 1); exit 1; }; }
+#else
+# test $cross_compiling = yes
+# { { echo $as_me:$LINENO: error: cannot check for file existence when cross compiling 5
+#echo $as_me: error: cannot check for file existence when cross compiling 2;}
+# { (exit 1); exit 1; }; }
if test -r $dir/include/readline/readline.h; then
eval $as_ac_File=yes
else
--- vdbeaux.c_orig 2005-09-17 19:48:56.0 +0200
+++ vdbeaux.c 2005-09-19 10:55:50.0 +0200
@@ -1509,7 +1509,11 @@
}
len = i = sqlite3VdbeSerialTypeLen(serial_type);
while( i-- ){
- buf[i] = (v0xFF);
+ if( serial_type==7 ){
+buf[(i+4)%8] = (v0xFF);
+ }else{
+buf[i] = (v0xFF);
+ }
v = 8;
}
return len;
@@ -1568,28 +1572,20 @@
pMem-flags = MEM_Int;
return 6;
}
-case 6: /* 8-byte signed integer */
+case 6: { /* 8-byte signed integer */
+ u64 x = (buf[0]24) | (buf[1]16) | (buf[2]8) | buf[3];
+ u32 y = (buf[4]24) | (buf[5]16) | (buf[6]8) | buf[7];
+ x = (x32) | y;
+ pMem-i = *(i64*)x;
+ pMem-flags = MEM_Int;
+ return 8;
+}
case 7: { /* IEEE floating point */
- u64 x;
- u32 y;
-#ifndef NDEBUG
- /* Verify that integers and floating point values use the same
- ** byte order. The byte order differs on some (broken) architectures.
- */
- static const u64 t1 = ((u64)0x3ff0)32;
- assert( 1.0==*(double*)t1 );
-#endif
-
- x = (buf[0]24) | (buf[1]16) | (buf[2]8) | buf[3];
- y = (buf[4]24) | (buf[5]16) | (buf[6]8) | buf[7];
+ u64 x = (buf[4]24) | (buf[5]16) | (buf[6]8)
Re: [sqlite] Issue with sqlite3_result_error inside aggregatefunction
You should not call sqlite3_result_* from within the step function of an aggregate. Those routines may be called from within the finalizer function only. Could you add some sort of assertion so that it is caught if the functions are called? So what should I do if an error occurs? Since I am ultimately running a user provided Python function there is always the possibility of an error happening. Roger
[sqlite] Using SQL and direct BTree interface to SQLite
Hello. I'm interested in using SQLite both at the SQL and BTree levels in an application. That is, I would like to create tables and be able to run queries using SQL while also inserting into, querying and updating tables directly via the BTree-level C API calls (thus bypassing the SQL subsystem entirely). Is there any information on how I can accomplish this without corrupting data, confusing SQLite, etc? If no documentation currently exists on this, I would be grateful for any pointers from those who are familiar with the guts of SQLite. Thanks, Brian
Re: [sqlite] Issue with sqlite3_result_error inside aggregatefunction
On Tue, 2005-09-27 at 08:33 -0700, Roger Binns wrote: You should not call sqlite3_result_* from within the step function of an aggregate. Those routines may be called from within the finalizer function only. So what should I do if an error occurs? Since I am ultimately running a user provided Python function there is always the possibility of an error happening. If an error occurs in a step function, record that fact in the aggregate context. Then when the finalizer is called, check the error flag in the context and call sqlite3_result_error at that point if it is appropriate to do so. -- D. Richard Hipp [EMAIL PROTECTED]
Re: [sqlite] Using SQL and direct BTree interface to SQLite
On Tue, 2005-09-27 at 09:36 -0600, Brian Roe wrote: I'm interested in using SQLite both at the SQL and BTree levels in an application. That is, I would like to create tables and be able to run queries using SQL while also inserting into, querying and updating tables directly via the BTree-level C API calls (thus bypassing the SQL subsystem entirely). Is there any information on how I can accomplish this without corrupting data, confusing SQLite, etc? The only documentation is comments in the code. Note that the BTree-layer interface is not a published API. It is subject to change in very incompatible ways without notice. Such changes may be semantic only - meaning that after upgrading SQLite you find that your application compiles and links fine but generates subtle and difficult to trace errors. The BTree interface is also fragile and is not designed for external use. It lacks much of the parameter checking that normally occurs on library routines since the Btree interface is not intended for that use. You will find that generating segfaults and corrupting your database files will be quite easy if you call the BTree layer directly. For these reasons, calls directly into the BTree layer are strongly discouraged. -- D. Richard Hipp [EMAIL PROTECTED]
Re: [sqlite] Using SQL and direct BTree interface to SQLite
D. Richard Hipp wrote: On Tue, 2005-09-27 at 09:36 -0600, Brian Roe wrote: I'm interested in using SQLite both at the SQL and BTree levels in an application. That is, I would like to create tables and be able to run queries using SQL while also inserting into, querying and updating tables directly via the BTree-level C API calls (thus bypassing the SQL subsystem entirely). Is there any information on how I can accomplish this without corrupting data, confusing SQLite, etc? The only documentation is comments in the code. Note that the BTree-layer interface is not a published API. It is subject to change in very incompatible ways without notice. Such changes may be semantic only - meaning that after upgrading SQLite you find that your application compiles and links fine but generates subtle and difficult to trace errors. The BTree interface is also fragile and is not designed for external use. It lacks much of the parameter checking that normally occurs on library routines since the Btree interface is not intended for that use. You will find that generating segfaults and corrupting your database files will be quite easy if you call the BTree layer directly. For these reasons, calls directly into the BTree layer are strongly discouraged. OK, excellent information; thank you. Let me rephrase the question slightly: I'm interesting in bypassing the SQL layer when possible, while not breaking compatibility with SQL. I am not specifically interesting in using BTree-layer calls, that just seemed to be the obvious next layer down below the SQL subsystem. Perhaps the use of EXPLAIN would show the way to implement certain types of common accesses I expect to be done frequently, such as inserting one row, selecting one row using a unique key or updating one row. Then I could correlate the virtual machine instructions with specific supported API calls? Meanwhile, SQL would still work for reporting or other types of ad-hoc queries. Or is there a better way? Thanks.
[sqlite] [OFF TOPIC] How to unsubscribe the mailing list?
Thanks, Cláudio Leopoldino __ Faça ligações para outros computadores com o novo Yahoo! Messenger http://br.beta.messenger.yahoo.com/
RE: [sqlite] Using SQL and direct BTree interface to SQLite
Perhaps the use of EXPLAIN would show the way to implement certain types of common accesses I expect to be done frequently, such as inserting one row, selecting one row using a unique key or updating one row. Then I could correlate the virtual machine instructions with specific supported API calls? Meanwhile, SQL would still work for reporting or other types of ad-hoc queries. Or is there a better way? Thanks. I think you'll find that you'd have to replicate so much of what's done after the SQL is compiled that there would be little (almost nothing) saved in terms of execution time and an immense increase in complexity. If you prepare a query and execute repeatedly there really isn't any penalty to using the SQL layer. -Tom
Re: [sqlite] Using SQL and direct BTree interface to SQLite
I was also looking to do something similar with tabular type data that exists in my application and create synthetic Sqlite tables without actually populating the Sqlite database. i.e., I just wanted to leverage the excellent SQL select engine of Sqlite to query the transient data that is already in RAM in my app (typically in large arrays). This would not only save RAM, but increase the overall speed of the program. But after looking at the Btree API, I concluded that making such a synthetic table hook was far from trivial. Such logic would have to be put between the VDBE layer and the Btree layer - a major code redesign. --- D. Richard Hipp [EMAIL PROTECTED] wrote: On Tue, 2005-09-27 at 09:36 -0600, Brian Roe wrote: I'm interested in using SQLite both at the SQL and BTree levels in an application. That is, I would like to create tables and be able to run queries using SQL while also inserting into, querying and updating tables directly via the BTree-level C API calls (thus bypassing the SQL subsystem entirely). Is there any information on how I can accomplish this without corrupting data, confusing SQLite, etc? The only documentation is comments in the code. Note that the BTree-layer interface is not a published API. It is subject to change in very incompatible ways without notice. Such changes may be semantic only - meaning that after upgrading SQLite you find that your application compiles and links fine but generates subtle and difficult to trace errors. The BTree interface is also fragile and is not designed for external use. It lacks much of the parameter checking that normally occurs on library routines since the Btree interface is not intended for that use. You will find that generating segfaults and corrupting your database files will be quite easy if you call the BTree layer directly. For these reasons, calls directly into the BTree layer are strongly discouraged. -- D. Richard Hipp [EMAIL PROTECTED] __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
[sqlite] csv .import problems
2 problems when importing data in sqlite3 with .import: 1. It leaves quotes in. If data field is enclosed in quotes, they are 'doubled'. (As reported in this list on Feb 16th) 2. When a string contains a comma, it is interpreted as separator, and an error is issued, of the type -4 fields were expected, and there were 6 There is a workaround. For (1), taking all quotes out with a tect processor, and .importing in .mode list. But not easy if some string data contain a comma, since then format gets corrupted. For (2), change the separator character to something else such as | or TAB. But it is a pity that csv format is not correctly read in. Anyway, that happens with most database managers I've seen. Antoni
Re: [sqlite] any suggestion for the database file extension?mine is xxx.drh
I recommend not using SDB (on windows machines) SQLite DataBase sounded like a good name too :) That is the default extension for something called Appfix Package and in certain circumstances, Windows will automatically back the DB up every time it is changed thinking that a DLL or Application has changed so the System Restore can roll it back at a later date
RE: [sqlite] csv .import problems
Here's a little fix I just wrote for an application for a custom format import. The incoming file is a .csv generated by MickeySoft Excel. The code below will scan a text buffer S returning a parsed field value. You must maintain a global referenced variable RP for retaining the current buffer position between calls, or change the function call to pass it. Unfortunately for some, the code is Borland Object Pascal. But, hey, any good programmer can write C in any language. Warning: This is just finished code that has been alpha tested only. And it is Quick and dirty, so I expect a couple of hundred messages containing solutions of two lines of code or less solving the same problem :-) Fred --- function ExtractData(S: string): string; var j: Integer; FS: String; begin FS := ''; if (RP = 1) and (S[RP] = FSeprator) then Inc(RP) else begin if S[RP] = FSeprator then Inc(RP); if S[RP] = FDelimiter then begin Inc(RP); repeat begin for j := RP to Length(S) do if S[j] = FDelimiter then break; if (S[j] = FDelimiter) and (S[j+1] = FDelimiter) then Inc(j); FS := FS + Copy(S, RP, j - RP); Inc(j); RP := j; end until (S[j] = FSeprator) or (j = Length(S)); end else begin if s[RP+1] = FSeprator then Inc(RP) else begin for j := RP to Length(S) do if S[j] = FSeprator then break; FS := FS + Copy(S, RP, j - RP); end; RP := j; end; end; Result := FS; end; - -Original Message- From: Antonio Francino [mailto:[EMAIL PROTECTED] Sent: Tuesday, September 27, 2005 7:13 PM To: sqlite-users@sqlite.org Subject: [sqlite] csv .import problems 2 problems when importing data in sqlite3 with .import: 1. It leaves quotes in. If data field is enclosed in quotes, they are 'doubled'. (As reported in this list on Feb 16th) 2. When a string contains a comma, it is interpreted as separator, and an error is issued, of the type -4 fields were expected, and there were 6 There is a workaround. For (1), taking all quotes out with a tect processor, and .importing in .mode list. But not easy if some string data contain a comma, since then format gets corrupted. For (2), change the separator character to something else such as | or TAB. But it is a pity that csv format is not correctly read in. Anyway, that happens with most database managers I've seen. Antoni
Re: [sqlite] Schemas
On 9/27/05, Vivien Malerba [EMAIL PROTECTED] wrote: Hi! I'm working on the SQLite interface between the Libgda library and the SQLite database. For the job, I need to know information about the database schema, and I'm using the PRAGMA statements regarding schemas. However I can't find the information about the AUTO INCREMENT attribute of a column. Can anyone help me? here they are, http://sqlite.org/datatypes.html http://sqlite.org/datatype3.html Thanks a lot, Vivien -- Firman Wandayandi Never Dreamt Before (http://php.hm/~firman/)
