Re: [sqlite] using sqlite as a temporary database to process lots of data
If you are using Linux, you can use the /dev/shm. This is a memory resident file system. I use this and find it about 200 times faster than writing to disk. The only problem is that this is erased when the server reboots. I hope this is of some use :) Ben Lorenzo Jorquera wrote: Hi, My company is starting a project in wich we will have to process large amounts of data and save the results to a db2 database. Because this data process needs to be very fast, the idea is to load the data in memory and apply the transformations in C++. I think that sqlite inmemory databases could be a good alternative to this, but when I proposed this solution to my boss, he asked if someone has already used sqlite for this purpose. I found that this use of sqlite is cited in the wiki, but I can't find any concrete examples. If any of you can share some success histories with me, I will be very grateful. Thanks in advance, Lorenzo. PD: please, forgive my english, it is a little rusty...
Re: [sqlite] using sqlite as a temporary database to process lots of data
If you are running Windows you may use a similar memory-mapped file (see the Win32API for details). Does anyone know which is fastest -- Using a memory-mapped file (or /dev/shm) in comparison with sqlite's internal memory mapped tree? -- Best regards / Med venlig hilsen Rasmus Christian Kaae - [EMAIL PROTECTED] or [EMAIL PROTECTED] Web: www.kalchas.dk www.eurojumping.com www.hestebasen.com Office: E4-119, Aalborg University. Mobile: +45 28 72 04 13 Quoting Ben Clewett [EMAIL PROTECTED]: If you are using Linux, you can use the /dev/shm. This is a memory resident file system. I use this and find it about 200 times faster than writing to disk. The only problem is that this is erased when the server reboots. I hope this is of some use :)
[sqlite] Error : Library routine called out of sequence
Hello there, I am facing one weird problem using SQLite. Many times I get an error Library routine called out of sequence while executing query Create table SOURCEFEED (SOURCEFEED,STATUS) values ( ABC,0) , I'm not getting the reason behind this. Why am I getting this error? My application is a multithread application and executes query from a worker thread. Let me know the possible solutions. Regards Ajay Sonawane
Re: [sqlite] Error : Library routine called out of sequence
I am facing one weird problem using SQLite. Many times I get an error Library routine called out of sequence while executing query Create table SOURCEFEED (SOURCEFEED,STATUS) values ( ABC,0) , I'm not getting the reason behind this. Why am I getting this error? My application is a multithread application and executes query from a worker thread. Are you trying to create a table or insert values into an existing table? I've never seen anything in SQL that suggests that you can do what you appear to be trying to do with that query, i.e. insert rows into a table while creating it.
Re: [sqlite] Error : Library routine called out of sequence
On Jun 28, 2005, at 5:08 AM, Ajay wrote: I am trying to insert values into table and executing that query from thread. There are 2 more threads that accesses same table. Do all the threads use the same sqlite handle? I believe that can cause problems.
RE: [sqlite] Error : Library routine called out of sequence
Yaa You are right; I solved the problem by using some of synchronization objects for thread access. Thanks for giving me direction. -Original Message- From: Will Leshner [mailto:[EMAIL PROTECTED] Sent: Tuesday, June 28, 2005 7:07 PM To: sqlite-users@sqlite.org Subject: Re: [sqlite] Error : Library routine called out of sequence On Jun 28, 2005, at 5:08 AM, Ajay wrote: I am trying to insert values into table and executing that query from thread. There are 2 more threads that accesses same table. Do all the threads use the same sqlite handle? I believe that can cause problems.
Re: [sqlite] using sqlite as a temporary database to process lots of data
On Tue, 28 Jun 2005, Rasmus Christian Kaae wrote: If you are running Windows you may use a similar memory-mapped file (see the Win32API for details). Does anyone know which is fastest -- Using a memory-mapped file (or /dev/shm) in comparison with sqlite's internal memory mapped tree? Haven't tested it, so take with a pinch of salt. /disclaimer My guess is that the SQLite internel memory only representation will be quicker, as there is less data transformation being done. The SQLite disk image must be written in such a way that is n-bit and bi-endian clean, so that the file can be read on any platform. This doesn't come for free. The in-memory database, however, just uses native pointers and datatypes of the processor, so is much quicker to manipulate and traverse. If you're not keeping the data, the in-memory is almost certainly the best way to go. If in doubt, test it. As to the original question, I can't say I've ever used SQLite that way, but only because I want to keep my data as is. -- Best regards / Med venlig hilsen Rasmus Christian Kaae - [EMAIL PROTECTED] or [EMAIL PROTECTED] Web: www.kalchas.dk www.eurojumping.com www.hestebasen.com Office: E4-119, Aalborg University. Mobile: +45 28 72 04 13 Quoting Ben Clewett [EMAIL PROTECTED]: If you are using Linux, you can use the /dev/shm. This is a memory resident file system. I use this and find it about 200 times faster than writing to disk. The only problem is that this is erased when the server reboots. I hope this is of some use :) -- /\ \ /ASCII RIBBON CAMPAIGN - AGAINST HTML MAIL X - AGAINST MS ATTACHMENTS / \
[sqlite] disk image malformed
Hello, Just wondering why I get this error when trying to open the DB with the Binary interface. It has actually worked on the odd occasion. sqlite l_main.dbl Unable to open database l_main.dbl: database disk image is malformed Regards, Leander
Re: [sqlite] disk image malformed
I recently reported a similar error. After upgrading to sqlite3, I noticed that I would occassionally get the database disk image is malformed errors using the library code. I just closed/reopened the db, retried the query and always had success on the retry. No idea why - D. Richard Hipp says that it's not possible for a database to be temporarily corrupt. I've also sometimes observed a some_table doesn't exist error. Closing/reopening and retrying the query also always works. I think that it's a good idea to have a wrapper around the sqlite library that will retry queries a few times if they don't return SQLITE_OK. This is especially useful in the cases when the db is busy but the busy handler isn't called (because apparently that would cause deadlock). Kevin --- Leander Gillard [EMAIL PROTECTED] wrote: Hello, Just wondering why I get this error when trying to open the DB with the Binary interface. It has actually worked on the odd occasion. sqlite l_main.dbl Unable to open database l_main.dbl: database disk image is malformed Regards, Leander __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
Re: [sqlite] using sqlite as a temporary database to process lots of data
I am currently using SQLite to process @ 400 million records (and climbing) a day by reading files, importing them into SQLite, and summarizing. The summed data goes into Oracle. This is a production application that is very stable. Holding the data in SQLite in memory as opposed to a C struct reduced development time and makes it easy to change the summarization output by just modifying the SQL statements. If you think the data layouts may be changed in the future or even dynamic / defined on the fly then SQLite would be a good tool. - Original Message - From: Lorenzo Jorquera [EMAIL PROTECTED] To: sqlite-users@sqlite.org Sent: Monday, June 27, 2005 4:55 PM Subject: [sqlite] using sqlite as a temporary database to process lots of data Hi, My company is starting a project in wich we will have to process large amounts of data and save the results to a db2 database. Because this data process needs to be very fast, the idea is to load the data in memory and apply the transformations in C++. I think that sqlite inmemory databases could be a good alternative to this, but when I proposed this solution to my boss, he asked if someone has already used sqlite for this purpose. I found that this use of sqlite is cited in the wiki, but I can't find any concrete examples. If any of you can share some success histories with me, I will be very grateful. Thanks in advance, Lorenzo. PD: please, forgive my english, it is a little rusty...
[sqlite] library and bin different
It looks as if the sys admin installed the libray with sqite 3 and perls DBD::Sqlite but used the Sqlite bin version instead of Sqlite 3.whatever for manipulating it from the shell. I'll ask him to install the sqlite3 bin files and see if it fixes itself. However it really does work every once in awhile from the shell. -- Leander
