Re: [sqlite] multiple writers for in-memory datastore

2008-04-21 Thread James Gregurich
interesting. thanks for the tip. I"ll give it some consideration. -James On Apr 21, 2008, at 1:07 :50PM, Scott Hess wrote: > If you create a file on disk and set PRAGMA synchronous = OFF, you > should get pretty close to the performance of a shared in-memory > database on most modern desktop

Re: [sqlite] multiple writers for in-memory datastore

2008-04-21 Thread Scott Hess
If you create a file on disk and set PRAGMA synchronous = OFF, you should get pretty close to the performance of a shared in-memory database on most modern desktop operating systems - maybe close enough that you won't care to do anything beyond that. If you further look at the recent discussion/pa

Re: [sqlite] multiple writers for in-memory datastore

2008-04-20 Thread James Gregurich
yes. However, CoreData queues up modified managed objects in a managed object context and then commits them all in one shot making sure the serialization is done on the back side. So, it does basically what someone here recommended earlier. I just don't have to write the mechanism myself.

Re: [sqlite] multiple writers for in-memory datastore

2008-04-20 Thread Dennis Cote
James Gregurich wrote: > I think I will go with CoreData on MacOSX and figure out something > else to do on Windows later. > > > You do know that CoreData uses SQLite for its persistant storage. Dennis Cote ___ sqlite-users mailing list sqlite-users

Re: [sqlite] multiple writers for in-memory datastore

2008-04-20 Thread Dan
On Apr 20, 2008, at 12:29 AM, James Gregurich wrote: > > oh good! That isn't the version that ships with Leopard, but I can > live with deploying my own version as part of my app. > > Will l get the writer parallelism I'm after as long as each thread > writes exclusively into its own attached db?

Re: [sqlite] multiple writers for in-memory datastore

2008-04-19 Thread James Gregurich
for those who may be interested: I ran a test with SQLite version: 3.5.8 I tried the scheme described earlier with each thread sharing a connection but writing into its own attached in-memory db on that connection. Didn't work. all but the first writer thread failed with a SQLITE_ERROR

Re: [sqlite] multiple writers for in-memory datastore

2008-04-19 Thread James Gregurich
gt; To: General Discussion of SQLite Database > Subject: Re: [sqlite] multiple writers for in-memory datastore > > > I don't immediately see how that would solve the problem. > > The limitation of interest here (based on my perhaps limited > understanding) is that loc

Re: [sqlite] multiple writers for in-memory datastore

2008-04-19 Thread Virgilio Alexandre Fornazin
Behalf Of James Gregurich Sent: sábado, 19 de abril de 2008 17:02 To: General Discussion of SQLite Database Subject: Re: [sqlite] multiple writers for in-memory datastore I don't immediately see how that would solve the problem. The limitation of interest here (based on my perhaps li

Re: [sqlite] multiple writers for in-memory datastore

2008-04-19 Thread James Gregurich
I don't immediately see how that would solve the problem. The limitation of interest here (based on my perhaps limited understanding) is that locking has file-level granularity. I don't immediately see how a VST implementation would allow for changing the locking granularity of the overall

Re: [sqlite] multiple writers for in-memory datastore

2008-04-19 Thread Virgilio Fornazin
what about creating a VFS for such task ? Can be accomplished in many ways, using heap memory, shared memory... not so easy to do, but not much complicated too... locking can be provided by multiple-readers single-writers locks strategies, etc... On Sat, Apr 19, 2008 at 2:29 PM, James Gregurich <[

Re: [sqlite] multiple writers for in-memory datastore

2008-04-19 Thread James Gregurich
oh good! That isn't the version that ships with Leopard, but I can live with deploying my own version as part of my app. Will l get the writer parallelism I'm after as long as each thread writes exclusively into its own attached db? in other wordstwo bulk insert operations going on simu

Re: [sqlite] multiple writers for in-memory datastore

2008-04-19 Thread Dan
On Apr 19, 2008, at 6:06 AM, James Gregurich wrote: > > I'll ask this question. The answer is probably "no," but I'll ask it > for the sake of completeness. > > > Suppose I created an in-memory db. I use the attach command to > associate an additional in-memory db. Suppose I assign the main db to

Re: [sqlite] multiple writers for in-memory datastore

2008-04-18 Thread James Gregurich
I'll ask this question. The answer is probably "no," but I'll ask it for the sake of completeness. Suppose I created an in-memory db. I use the attach command to associate an additional in-memory db. Suppose I assign the main db to thread 1 and the associated db to thread 2. Can I share th

Re: [sqlite] multiple writers for in-memory datastore

2008-04-18 Thread Ken
How about this instead. Read your records, parse and format into some known format by your application. Write the data to disk in a file. Then put a single entry into a sqlite table. specifing the on disk file name. Sqlite may only have one write operation running concurrently. There are no

Re: [sqlite] multiple writers for in-memory datastore

2008-04-18 Thread James Gregurich
On Apr 18, 2008, at 2:33 :32PM, Dennis Cote wrote: > > To share an attached database the threads must be able to name it, and > this is only possible with a file database. you could change the open() function to be able to assign a name to an in-memory db and then keep a mapping of all the name

Re: [sqlite] multiple writers for in-memory datastore

2008-04-18 Thread Dennis Cote
James Gregurich wrote: >> You will have to open the memory database and attach the db file since >> SQLite can't attach to a memory database. > > is this information wrong? > http://www.blitzbasic.com/Community/posts.php?topic=60981 > No it's not. I was mistaken. SQLite can attach a memory dat

Re: [sqlite] multiple writers for in-memory datastore

2008-04-18 Thread James Gregurich
On Apr 18, 2008, at 1:25 :36PM, Dennis Cote wrote: > James Gregurich wrote: >> >> suppose I create a temporary db file on disk. Each task ( a thread) >> opens a connection to the temp file and attaches an in-memory db to >> it. > > You will have to open the memory database and attach the db file

Re: [sqlite] multiple writers for in-memory datastore

2008-04-18 Thread Dennis Cote
James Gregurich wrote: > > suppose I create a temporary db file on disk. Each task ( a thread) > opens a connection to the temp file and attaches an in-memory db to > it. You will have to open the memory database and attach the db file since SQLite can't attach to a memory database. I woul

Re: [sqlite] multiple writers for in-memory datastore

2008-04-18 Thread James Gregurich
I'm working on a commercial, boxed, desktop product. I can't be creating new mounted disks on a customer's system every time he uses my application. How about this... suppose I create a temporary db file on disk. Each task ( a thread) opens a connection to the temp file and attaches an

Re: [sqlite] multiple writers for in-memory datastore

2008-04-18 Thread Dennis Cote
James Gregurich wrote: > If the sqlite statement had a temporary storage area so that I could > load up a bunch of rows and then commit them in one shot so that the > lock on the db was not held very long by a single transaction, that > would probably work. > Using a RAM disk you could inse

Re: [sqlite] multiple writers for in-memory datastore

2008-04-18 Thread James Gregurich
If the sqlite statement had a temporary storage area so that I could load up a bunch of rows and then commit them in one shot so that the lock on the db was not held very long by a single transaction, that would probably work. However, my reading of the documentation leads me to believe tha

Re: [sqlite] multiple writers for in-memory datastore

2008-04-18 Thread John Stanton
Just use a thread as a DB handler. Queue transactions to it using some IPC mechanism like a message queue or named pipe. Another way would be to synchronize access to the DB handle using a mutex. James Gregurich wrote: > hi! > > > I need to set up multiple writers to an in-memory datastore.

Re: [sqlite] multiple writers for in-memory datastore

2008-04-18 Thread Dennis Cote
James Gregurich wrote: > > I need to set up multiple writers to an in-memory datastore. I just > discovered that you can't have more than one connection to an in- > memory store. > > I can give each task its own independent datastore if there is a way I > can merge the contents of each store

[sqlite] multiple writers for in-memory datastore

2008-04-18 Thread James Gregurich
hi! I need to set up multiple writers to an in-memory datastore. I just discovered that you can't have more than one connection to an in- memory store. I can give each task its own independent datastore if there is a way I can merge the contents of each store into a central store. Is there