Re: [sqlite] Sporadic freezes of sqlite
Unfortunately these two options are not for me. I'm not in control of servers hardware, so my application should work on the given servers and nobody will adjust them for my application. And about first option, I believe what you say is to use in-memory database for intensive operations. But all my application consists of these intensive operations alone. So that you can reasonably argue that I should reject the idea of on-disk database and work totally in memory. And i can agree with you. But there's a couple of requirements that make things difficult. And the main of it is application should have some durability and survive power outages, crashes and reboots. "Some" because I can sacrifice for example everything that was written up to 5 minutes before power outage, but everything else should stay. And at this point all idea of in-memory database is ruined and I have to cope somehow with problems of frequent writings to disk. Pavel On Fri, Apr 24, 2009 at 6:13 AM, January Weinerwrote: >> I have an application written using sqlite. It writes into the >> database very intensively. And I noticed that it works nice and very >> fast but from time to time it just freezes for several seconds (I've >> registered freezes up to 8 secs). After some tracing of sqlite code > > I had the same problem. Also, it was increasing with database size. > Depending on your environment, work procedure and whether you want > speed or security, there are two things that work beautifully for me: > > 1) do the intentsive work on a db copy that sits on a ramdisk (or > tmpfs filesystem). I do that if I have to create a new database or > rebuild this from scratch, and since the process is supervised, there > is not really a danger of data loss. > > 2) for normal operation, I use a software RAID from flash disks, which > is not as fast (for data transfer) as a hard drive or SSD, but it is > has a response time better by an order of magnitude (at least) than > even a good hard drive. > > j. > > -- > -Dr. January Weiner 3 -+--- > Inst. of Bioinformatics, UKM, Univ. of Muenster | Von-Esmarch-str. 54 > (+49) (251) 83 53002 | D48149 Münster > http://www.compgen.uni-muenster.de/ | Germany > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] TEMP_STORE not working ?(3.5.9)
After code reading, I have discovered that the culprit is the statement journal. Regardless of TEMP_STORE = MEMORY or not, the statement journal, which is treated differently than a regular journal file, is placed in a temporary file. This effectively kills the usefulness and purpose of the TEMP_STORE directive. Kris Groves wrote: > Hi, > > Just want to bump this, I really need to get to the bottom of this. > > Thanks for any info, > Kris. > > Kris Groves wrote: > >> Hi, >> From what I understand : >> - default behavior is to use files for temporary stuff. >> - the directory that will be used for these temporary files can be >> defined via pragma (temp_store_directory). If the pragma is not used, >> it will default to the first hardcoded directory (linux), in the order >> that follows: /var/tmp, /usr/tmp, /tmp, or finally current directory. >> >> So, in the environment I am running in, either those directories do not >> exist, or are not writable to the user under which the process is >> running. The result being an "error 14: unable to open database file" >> as soon as temporary files are needed. >> >> After a little digging I discover SQLITE_TEMP_STORE compilation flag. >> So I export CFLAGS=-DSQLITE_TEMP_STORE=3, run configure and remake, >> figuring that the temp files will now reside in memory, and need no >> writing into a directory. However, the problem remains. >> >> When I look through the code, there is no instance of SQLITE_TEMP_STORE, >> only TEMP_STORE... So I repeat the above with -DTEMP_STORE. Same result. >> >> Then I add a path that I know is accessible to the user under which the >> process runs, to the azDirs array in the unixGetTempname function. >> Voila.. working now.. >> >> I've retested with default TEMP_STORE and TEMP_STORE compiled in a >> 3(memory only). And regardless of the setting, it only works if there >> is a readable/writable directory... >> >> I would think that if TEMP_STORE=3, then no directory is required ? Is >> this a bug, or am I misunderstanding something ? >> >> Thanks, >> Kris. >> >> ___ >> sqlite-users mailing list >> sqlite-users@sqlite.org >> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users >> >> > > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] .genfkey & quoted table names
Hi, I found a bug in the .genfkey functionality: If a foreign key constraint references a table with a quoted table name, the .genfkey command will fail on that constraint. Example: create table "t.3"(c1 primary key); create table t13(c1, foreign key(c1) references "t.3"(c1)); .genfkey yields: Error in table t13: foreign key columns do not exist I looked at the code in shell.c but I could not find the right place to fix this yet. Does anyone have a suggestion on fixing this bug? Thomas -- Dipl.-Ing. Thomas Henlich Liebigstr. 38, 01187 Dresden tel.: +49 [0]351 4712357 mobil: +49 [0]175 1723884 ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] record order in select
> If "order by" isn't used in a select statment, does the result records > ordered in rowid? If you omit an ORDER BY clause, the order of the resulting data is undefined. It might happen to be ordered by rowid in some cases, but this is not guaranteed. (Might change in a future version, etc.) If you need to ensure a particular order, use an ORDER BY clause. The good news is that if you order by rowid / PRIMARY KEY, you likely won't see much (if any) performance drop, but you're guaranteed to get the order you want. The following "reverse_unordered_selects" pragma may be of interest: http://sqlite.org/pragma.html#pragma_reverse_unordered_selects ~Eric ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] the speed of embedded database engines, sqlite3 vs berkeley db, I'm confused
On Fri, Apr 24, 2009 at 5:27 AM, John Stantonwrote: > Sqlte provides COMMIT and ROLLBACK unlike Berkeley. If you can get by > without the advanced features of Sqlite, then use Berkely and take > advantage of its simplicity and faster execution. BDB does support transactions... http://www.oracle.com/technology/documentation/berkeley-db/xml/gsg_xml_txn/cxx/usingtxns.html In fact, BDB acts as the (a) transactional layer in MySQL (the alternative is InnoDB). Of course, other advantages of SQLite still apply. A key-value metaphor can only be pushed so far. > > Note that for best performance an Sqlite application should group > database inserts, deletes and updates into transactions. > > liubin liu wrote: >> our project is using sqlite3, but found that the speed is too slow. >> I just tested the BerkeleyDB, and found that the speed is very fast. But I >> knew the sqlite3 is fast enough. And so I'm confused. >> I may be using sqlite3 in wrong way? >> >> anyway, next is my test code. I'm glad to receive your message. >> >> __ >> >> >> // http://www.ibm.com/developerworks/cn/linux/l-embdb/ >> >> //head/// >> #include >> #include // for system >> #include // for memset strcpy >> #include // for time >> >> #include // for Sqlite3 >> #include // for Berkeley DB >> >> >> >> macro and struct >> / >> #define DB_FILE_SQLITE "test_sqlite_0.1.db" >> #define DB_FILE_BDB "test_bdb_0.1.db" >> >> struct customer >> { >> int c_id; >> char name[10]; >> char address[20]; >> int age; >> }; >> >> >> >> >> >> >> // global variable >> /// >> >> sqlite3 *db = NULL; >> >> int ret = -1; // 各函数返回值 >> >> >> >> >> >> >> >> // func proto >> /// >> >> void way01(); // 打开、关闭的影响 >> >> >> >> / sqlite3 // >> >> int sqlite_createtb(sqlite3 *db); >> int sqlite_insertdb(sqlite3 *db); >> >> int getdata_sqlite ( sqlite3 *db, struct customer *tb1 ); >> >> >> >> /// berkeley db >> >> int bdb_createdb(); // 包含了插入数据 >> >> void print_error(int r); >> void init_dbt( DBT *key, DBT *data ); >> >> >> >> >> >> >> /// code/// >> >> int main ( void ) >> { >> int c = 0; >> >> system ( "rm -rf test_0.1.db" ); >> ret = sqlite3_open ( DB_FILE_SQLITE, ); >> ret = sqlite_createtb(db); >> ret = sqlite_insertdb(db); >> sqlite3_close (db); >> >> printf ( "Sqlite3 / Berkeley DB, 创建数据库 + 插入数据 ... 完成\n" ); >> >> printf ( "/\n" ); >> printf ( "1 : 查询性能比较 - Berkeley DB 与 Sqlite3 ///\n" ); >> >> while ( (c=getchar()) != 'q' ) >> { >> switch (c) >> { >> case '1': >> way01(); >> break; >> default: >> break; >> } >> } >> >> system ( "rm -rf test_sqlite_0.1.db" ); >> system ( "rm -rf test_bdb_0.1.db" ); >> >> return 0; >> } >> >> /// >> // 查询性能比较 - Berkeley DB 与 Sqlite3 >> void way01() >> { >> time_t tick1, tick2; >> >> int i = 0; >> int num = 1000*100; >> >> struct customer tb_data; >> >> /// >> time ( ); >> for ( i=0; i > { >> ret = sqlite3_open ( DB_FILE_SQLITE, ); >> ret = getdata_sqlite ( db, _data ); >> sqlite3_close (db); >> } >> time ( ); >> printf("Sqlite3 : 打开、关闭并操作数据库文件 %d 次, 时间为: %4ld s\n", num, tick2 - >> tick1 ); >> >> /// >> bdb_createdb(); >> } >> >> >> >> >> >> >> >> /// >> void *callback(void *para, int col, char **value, char **colname ) >> { >> //int i; >> //for(i=0; i > //printf("%s, ", (value[i] ? value[i] : "NULL") ); >> //} >> //printf("col = %d\n", col); >> return (void *) 0; >> } >> int getdata_sqlite ( sqlite3 *db, struct customer *tb1 ) >> { >> char *sql = "SELECT * FROM table1 WHERE id=500;"; >> ret = sqlite3_exec ( db, sql, (void *)callback, NULL, NULL ); >> return 0; >> } >> >> /// >> int sqlite_createtb( sqlite3 *db ) >> { >> char *sql1 = "CREATE TABLE table1 (id INTEGER, name VARCHAR(10), >> address >> VARCHAR(20), age INTEGER)"; >> ret = sqlite3_exec ( db, sql1, NULL, NULL, NULL ); >> return 0; >> } >> int
Re: [sqlite] record order in select
"Wenton Thomas"wrote in message news:790786.83985...@web111003.mail.gq1.yahoo.com > If "order by" isn't used in a select statment, does the result > records ordered in rowid? Generally, no. It could accidentally happen this way, but there's no guarantee. If you want a particular order (by rowid or otherwise), request it with ORDER BY clause. Igor Tandetnik ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] the speed of embedded database engines, sqlite3 vs berkeley db, I'm confused
Sqlte provides COMMIT and ROLLBACK unlike Berkeley. If you can get by without the advanced features of Sqlite, then use Berkely and take advantage of its simplicity and faster execution. Note that for best performance an Sqlite application should group database inserts, deletes and updates into transactions. liubin liu wrote: > our project is using sqlite3, but found that the speed is too slow. > I just tested the BerkeleyDB, and found that the speed is very fast. But I > knew the sqlite3 is fast enough. And so I'm confused. > I may be using sqlite3 in wrong way? > > anyway, next is my test code. I'm glad to receive your message. > > __ > > > // http://www.ibm.com/developerworks/cn/linux/l-embdb/ > > //head/// > #include > #include // for system > #include // for memset strcpy > #include // for time > > #include // for Sqlite3 > #include // for Berkeley DB > > > > macro and struct > / > #define DB_FILE_SQLITE "test_sqlite_0.1.db" > #define DB_FILE_BDB "test_bdb_0.1.db" > > struct customer > { > int c_id; > char name[10]; > char address[20]; > int age; > }; > > > > > > > // global variable > /// > > sqlite3 *db = NULL; > > int ret = -1; // 各函数返回值 > > > > > > > > // func proto > /// > > void way01(); // 打开、关闭的影响 > > > > / sqlite3 // > > int sqlite_createtb(sqlite3 *db); > int sqlite_insertdb(sqlite3 *db); > > int getdata_sqlite ( sqlite3 *db, struct customer *tb1 ); > > > > /// berkeley db > > int bdb_createdb(); // 包含了插入数据 > > void print_error(int r); > void init_dbt( DBT *key, DBT *data ); > > > > > > > /// code/// > > int main ( void ) > { > int c = 0; > > system ( "rm -rf test_0.1.db" ); > ret = sqlite3_open ( DB_FILE_SQLITE, ); > ret = sqlite_createtb(db); > ret = sqlite_insertdb(db); > sqlite3_close (db); > > printf ( "Sqlite3 / Berkeley DB, 创建数据库 + 插入数据 ... 完成\n" ); > > printf ( "/\n" ); > printf ( "1 : 查询性能比较 - Berkeley DB 与 Sqlite3 ///\n" ); > > while ( (c=getchar()) != 'q' ) > { > switch (c) > { > case '1': > way01(); > break; > default: > break; > } > } > > system ( "rm -rf test_sqlite_0.1.db" ); > system ( "rm -rf test_bdb_0.1.db" ); > > return 0; > } > > /// > // 查询性能比较 - Berkeley DB 与 Sqlite3 > void way01() > { > time_t tick1, tick2; > > int i = 0; > int num = 1000*100; > > struct customer tb_data; > > /// > time ( ); > for ( i=0; i{ > ret = sqlite3_open ( DB_FILE_SQLITE, ); > ret = getdata_sqlite ( db, _data ); > sqlite3_close (db); > } > time ( ); > printf("Sqlite3 : 打开、关闭并操作数据库文件 %d 次, 时间为: %4ld s\n", num, tick2 - > tick1 ); > > /// > bdb_createdb(); > } > > > > > > > > /// > void *callback(void *para, int col, char **value, char **colname ) > { > //int i; > //for(i=0; i //printf("%s, ", (value[i] ? value[i] : "NULL") ); > //} > //printf("col = %d\n", col); > return (void *) 0; > } > int getdata_sqlite ( sqlite3 *db, struct customer *tb1 ) > { > char *sql = "SELECT * FROM table1 WHERE id=500;"; > ret = sqlite3_exec ( db, sql, (void *)callback, NULL, NULL ); > return 0; > } > > /// > int sqlite_createtb( sqlite3 *db ) > { > char *sql1 = "CREATE TABLE table1 (id INTEGER, name VARCHAR(10), address > VARCHAR(20), age INTEGER)"; > ret = sqlite3_exec ( db, sql1, NULL, NULL, NULL ); > return 0; > } > int sqlite_insertdb(sqlite3 *db) > { > time_t tick1, tick2; > int i = 0; > int num = 1000; > > char *qf = "INSERT INTO table1 VALUES (%d, %Q, %Q, %d)"; > char *sql = NULL; > > time ( ); > sqlite3_exec ( db, "BEGIN", NULL, NULL, NULL ); > for (i=0;i { > sql = sqlite3_mprintf ( qf, i, "javer", "chengdu", 32*i ); > ret = sqlite3_exec(db, sql, NULL, NULL, NULL); > sqlite3_free (sql); > } >
Re: [sqlite] Sporadic freezes of sqlite
> I have an application written using sqlite. It writes into the > database very intensively. And I noticed that it works nice and very > fast but from time to time it just freezes for several seconds (I've > registered freezes up to 8 secs). After some tracing of sqlite code I had the same problem. Also, it was increasing with database size. Depending on your environment, work procedure and whether you want speed or security, there are two things that work beautifully for me: 1) do the intentsive work on a db copy that sits on a ramdisk (or tmpfs filesystem). I do that if I have to create a new database or rebuild this from scratch, and since the process is supervised, there is not really a danger of data loss. 2) for normal operation, I use a software RAID from flash disks, which is not as fast (for data transfer) as a hard drive or SSD, but it is has a response time better by an order of magnitude (at least) than even a good hard drive. j. -- -Dr. January Weiner 3 -+--- Inst. of Bioinformatics, UKM, Univ. of Muenster | Von-Esmarch-str. 54 (+49) (251) 83 53002| D48149 Münster http://www.compgen.uni-muenster.de/ | Germany ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] record order in select
If "order by" isn't used in a select statment, does the result records ordered in rowid? ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Sporadic freezes of sqlite
"D. Richard Hipp"schrieb im Newsbeitrag news:fb9c47d5-1b72-4574-bb44-4f0d6685f...@hwaci.com... > FWIW, we are in the process of "productizing" the > test_async.c asynchronous VFS for SQLite. > The new async VFS might be available as a compile-time > option or as a loadable extension on both windows and > unix in 3.6.14. > The async VFS does all disk writes in a background thread... > ... > The async VFS also uses more memory, since the data > waiting to be written to disk has to be stored somewhere. Not that much of a problem nowadays (at least if not running on an embedded device)... > It might use a lot more memory if you are committing changes > to the database faster than the disk and the background writer > thread can handle them. That's understandable - and Ok - but leads to some questions: (assuming we are running on windows as in my case) How's the "read-uncommitted"-case handled then in the main-thread of the hosting process - is there an "automatic cross-thread-linking" to the "yet to write to the disk"-pages and the Selects already get (merge) their data from these pages? And how would we have to handle the case, if the user simply is closing the Application-process gracefully (for example by closing the Main-Window) - but your writerqueue yet contains outstanding writejobs? Is there some "state-info" which we could check - and set the process "on-hold" by ourselfs until all writes are finished? Is there only one vfs-WriterQueue-Thread per process - or is it per Connection (DB-Handle) - I assume per process, but just to make sure... In case there is just one WriterQueue-Thread per process - is there any chance, that the current backup-implementation could make use of that central writer-thread? I mean, in case of the more granular working backup-mode, which automatically retriggers itself in case of some writes on another DB-Handle (Connection) - we can currently not use this "low-prio"-backupmode on our AppServer, which holds "per-thread" DB-Connections in its WorkerThread-Pool. Currently, if we start an additional "low-prio" Backup- Thread - this thread will probably never finish - in case the other WorkerThreads perform writes against their (separate) SQLite-DB-ConnectionHandles. So, what we had planned for our AppServer was, to direct all DB-Write-Operations from the WorkerThreads to a dedicated DBWriterThread - which holds a separate DB-Handle too - but in case of a backup would pass its very same DB-Handle to the low-prio backupthread (thereby avoiding the retriggering). Do you see any chance (since all writes are already "centralized" if one uses the new async-writer-approach), that in case of a running "granular-backup" we could avoid our planned "workaround"? In either case - that will be a nice addition to the engine - thank you. Olaf Schmidt ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] the speed of embedded database engines, sqlite3 vs berkeley db, I'm confused
Thanks I amend the code according to your message. Marcus Grimm wrote: > > in your test code I see that you open the sqlite DB > each time you perform the single query. > I guess it would be fair to open the sqlite DB only once > outside your test loop. > 2nd: You query against ID without an index, try either > define an index on ID or change your table definition > into "ID INTEGER PRIMARY KEY" which will have an index > on ID automatically, I think.. > > Marcus > >> >> our project is using sqlite3, but found that the speed is too slow. >> I just tested the BerkeleyDB, and found that the speed is very fast. But >> I >> knew the sqlite3 is fast enough. And so I'm confused. >> I may be using sqlite3 in wrong way? >> >> anyway, next is my test code. I'm glad to receive your message. >> >> __ >> >> >> // http://www.ibm.com/developerworks/cn/linux/l-embdb/ >> >> //head/// >> #include >> #include // for system >> #include // for memset strcpy >> #include // for time >> >> #include // for Sqlite3 >> #include // for Berkeley DB >> >> >> >> macro and struct >> / >> #define DB_FILE_SQLITE "test_sqlite_0.1.db" >> #define DB_FILE_BDB "test_bdb_0.1.db" >> >> struct customer >> { >> int c_id; >> char name[10]; >> char address[20]; >> int age; >> }; >> >> >> >> >> >> >> // global variable >> /// >> >> sqlite3 *db = NULL; >> >> int ret = -1; // åå½æ°è¿åå¼ >> >> >> >> >> >> >> >> // func proto >> /// >> >> void way01(); // æå¼ãå ³éçå½±å >> >> >> >> / sqlite3 // >> >> int sqlite_createtb(sqlite3 *db); >> int sqlite_insertdb(sqlite3 *db); >> >> int getdata_sqlite ( sqlite3 *db, struct customer *tb1 ); >> >> >> >> /// berkeley db >> >> int bdb_createdb(); // å å«äºæå ¥æ°æ® >> >> void print_error(int r); >> void init_dbt( DBT *key, DBT *data ); >> >> >> >> >> >> >> /// code >> /// >> >> int main ( void ) >> { >> int c = 0; >> >> system ( "rm -rf test_0.1.db" ); >> ret = sqlite3_open ( DB_FILE_SQLITE, ); >> ret = sqlite_createtb(db); >> ret = sqlite_insertdb(db); >> sqlite3_close (db); >> >> printf ( "Sqlite3 / Berkeley DB, å建æ°æ®åº + æå ¥æ°æ® ... >> å®æ\n" ); >> >> printf ( "/\n" ); >> printf ( "1 : æ¥è¯¢æ§è½æ¯è¾ - Berkeley DB ä¸ Sqlite3 ///\n" ); >> >> while ( (c=getchar()) != 'q' ) >> { >> switch (c) >> { >> case '1': >> way01(); >> break; >> default: >> break; >> } >> } >> >> system ( "rm -rf test_sqlite_0.1.db" ); >> system ( "rm -rf test_bdb_0.1.db" ); >> >> return 0; >> } >> >> /// >> // æ¥è¯¢æ§è½æ¯è¾ - Berkeley DB ä¸ Sqlite3 >> void way01() >> { >> time_t tick1, tick2; >> >> int i = 0; >> int num = 1000*100; >> >> struct customer tb_data; >> >> /// >> time ( ); >> for ( i=0; i> { >> ret = sqlite3_open ( DB_FILE_SQLITE, ); >> ret = getdata_sqlite ( db, _data ); >> sqlite3_close (db); >> } >> time ( ); >> printf("Sqlite3 : æå¼ãå ³é并æä½æ°æ®åºæ件 %d 次, >> æ¶é´ä¸º: %4ld s\n", num, tick2 - >> tick1 ); >> >> /// >> bdb_createdb(); >> } >> >> >> >> >> >> >> >> /// >> void *callback(void *para, int col, char **value, char **colname ) >> { >> // int i; >> // for(i=0; i > // printf("%s, ", (value[i] ? value[i] : "NULL") ); >> // } >> // printf("col = %d\n", col); >> return (void *) 0; >> } >> int getdata_sqlite ( sqlite3 *db, struct customer *tb1 ) >> { >> char *sql = "SELECT * FROM table1 WHERE id=500;"; >> ret = sqlite3_exec ( db, sql, (void *)callback, NULL, NULL ); >> return 0; >> } >> >> /// >> int sqlite_createtb( sqlite3 *db ) >> { >> char *sql1 = "CREATE TABLE table1 (id INTEGER, name VARCHAR(10), address >> VARCHAR(20), age INTEGER)"; >> ret = sqlite3_exec ( db, sql1, NULL, NULL, NULL ); >> return 0; >> } >> int sqlite_insertdb(sqlite3 *db) >> { >> time_t tick1, tick2; >> int i = 0; >> int num = 1000; >> >> char *qf = "INSERT INTO table1 VALUES (%d, %Q, %Q, %d)"; >> char