On 16 Apr 2014, at 2:39am, Nick Eubank <nickeub...@gmail.com> wrote:
> -- am I going to have problems using all 16gb of ram on my Windows 8 > machine for data manipulations if I switch to SQLite? Or will SQLite set me > free? If you're talking about per-query memory cap, then that won't figure into anything SQLite does. SQLite never needs to fit an entire index or result-set into memory at one time. Only a single row or a chunk of rows is held in memory at one time. Worth remembering that SQLite is used mostly in small devices with extremely limited memory (smartphones, embedded devices), so it is designed not to be a memory-hog. The exception to the above is when you specifically declare an entire database or a table to be in memory rather than in a persistent storage device. Which is sometimes done for temporary tables or where something needs to be done extremely quickly. > -- Is there any reason I should NOT use SQLite for manipulation of large > datasets like this (for example, pulling out unique pairs of transaction > participants, averages across users, etc.)? All the literature I can find > talks about SQL database choices for people setting up databases that will > be queried by lots of people, and I just can't find any input for people > like me who just want a data manipulation tool for data that's too big to > read into RAM and manipulate with the usual suspects (R, Stata, Matlab, > etc.). If you're doing most of your work in R or Matlab (I've never used Stata) then you'll probably be using just a thin SQLite 'shim' that gets your data into and out of variables. It won't be a problem. >From your description it's worth doing the experiment to see whether it's good >for you. One of the reasons /not/ to use SQLite is that the user really needs >concurrent multi-user access which supports high-resolution locking, and you >don't have that problem. Others can be found in the second half of this >document: <http://sqlite.org/whentouse.html> I would recommend that you download the SQLite shell tool and get to know it a little. Not only is it useful in creating a SQLite database from an SQL command dump or CSV files, but it allows you to test whether SQLite understands your SQL commands or not (you might be using a trick that works only in Postgres). If something works in the shell tool but not in your own code, you know it's because of a bug in your own code. <http://sqlite.org/download.html> Simon. _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
