[sqlite] Using SQLite for write only workload, splitting databases into smaller files, but need to query it as a whole

2014-05-03 Thread Hayden Livingston
I have a situation where I store a data in a text file with tabs, and it's becoming really quite unwieldy to use this. My workload is quite straightforward, I have a multi-threaded application that logs to this file but from within any single thread at any given time. So from SQLite's perspective

Re: [sqlite] Using SQLite for write only workload, splitting databases into smaller files, but need to query it as a whole

2014-05-03 Thread Simon Slavin
On 3 May 2014, at 1:59pm, Hayden Livingston halivings...@gmail.com wrote: My workload is quite straightforward, I have a multi-threaded application that logs to this file but from within any single thread at any given time. So from SQLite's perspective only one person will be writing to this

Re: [sqlite] Using SQLite for write only workload, splitting databases into smaller files, but need to query it as a whole

2014-05-03 Thread Petite Abeille
On May 3, 2014, at 2:59 PM, Hayden Livingston halivings...@gmail.com wrote: Thoughts? Take a look at ‘ATTACH’, it might help: http://www.sqlite.org/lang_attach.html ___ sqlite-users mailing list sqlite-users@sqlite.org

Re: [sqlite] Using SQLite for write only workload, splitting databases into smaller files, but need to query it as a whole

2014-05-03 Thread Hayden Livingston
That's clever. I'm thinking how I'll go about this .. essentially the filename is devised by time splits. I could do what you're saying but then I can't move the file. Basically, the good part is that our warehousing department (which is what this is used for) can look at order inventories every 5

Re: [sqlite] Using SQLite for write only workload, splitting databases into smaller files, but need to query it as a whole

2014-05-03 Thread Hayden Livingston
This looks promising. I sooo wish it didn't have a limit to number of databases. But I think I could reasonably do something like coalesce the databases into a new database once every 2 hours. I also need to generate some code to figure out how to address the tables which I guess means I'll have

Re: [sqlite] Using SQLite for write only workload, splitting databases into smaller files, but need to query it as a whole

2014-05-03 Thread Petite Abeille
On May 3, 2014, at 3:40 PM, Hayden Livingston halivings...@gmail.com wrote: This looks promising. I sooo wish it didn't have a limit to number of databases. 10 by default if I recall properly. Can be perhaps be increased to 62 at most: http://www.sqlite.org/limits.html But I think I could

Re: [sqlite] Using SQLite for write only workload, splitting databases into smaller files, but need to query it as a whole

2014-05-03 Thread Hayden Livingston
Ok. But I'm wondering how GROUPBY and stuff will work? On Sat, May 3, 2014 at 7:13 AM, Petite Abeille petite.abei...@gmail.comwrote: On May 3, 2014, at 3:40 PM, Hayden Livingston halivings...@gmail.com wrote: This looks promising. I sooo wish it didn't have a limit to number of

Re: [sqlite] Using SQLite for write only workload, splitting databases into smaller files, but need to query it as a whole

2014-05-03 Thread Simon Slavin
On 3 May 2014, at 2:35pm, Hayden Livingston halivings...@gmail.com wrote: That's clever. I'm thinking how I'll go about this .. essentially the filename is devised by time splits. I could do what you're saying but then I can't move the file. Basically, the good part is that our warehousing

Re: [sqlite] Using SQLite for write only workload, splitting databases into smaller files, but need to query it as a whole

2014-05-03 Thread Teg
Hello Hayden, If I was doing this, I'd use a single database as the core then periodically from within the app, write out a new database with the last 5 minutes worth of data. Then let them run off the copy while you maintain a master copy of the DB. In that way this splitting and multiple DB