It is already wrapped in a transaction. I seem to remember seeing somewhere that the .import command doesn't understand escaping, e.g.
"one","two,three" will get imported as "one" | "two | three" (the quotes are part of the data, and the second column was split into two by the comma) Just a point to be aware of. On Tue, May 1, 2012 at 11:22 PM, Black, Michael (IS) <michael.bla...@ngc.com > wrote: > You need to try and do an import from the shell. GUIs seem to have way > too many limits. > > http://sqlite.org/download.html > > > > Don't do any indexes up front....do them afterwords if they'll help your > queries. Indexes will slow down your import notably. > > > > I don't think you're anywhere near the limits of sqlite since it talks > about terabytes. > > http://sqlite.org/limits.html > > > > Somebody else can answer for sure but wrapping your .import inside a > transaction may be a good thing. > > I don't know if that's done by default. > > > > Your queries are liable to be pretty slow depending on what you have to do. > > > > > > > > > > Michael D. Black > > Senior Scientist > > Advanced Analytics Directorate > > Advanced GEOINT Solutions Operating Unit > > Northrop Grumman Information Systems > > ________________________________ > From: sqlite-users-boun...@sqlite.org [sqlite-users-boun...@sqlite.org] > on behalf of peter korinis [kori...@earthlink.net] > Sent: Tuesday, May 01, 2012 3:06 PM > To: sqlite-users@sqlite.org > Subject: EXT :[sqlite] is SQLite the right tool to analyze a 44GB file > > I'm new to SQLite . not a programmer . not a DBA . just an end-user with no > dev support for a pilot project (single user, no updates, just queries). > > > > I want to analyze the data contained in a 44GB csv file with 44M rows x 600 > columns (fields all <15 char). Seems like a DBMS will allow me to query it > in a variety of ways to analyze the data. > > > > I have the data files and SQLite on my laptop: a 64-bit Win7 Intel > dual-proc > with 4GB RAM + 200GB free disk space. > > End-user tools like Excel & Access failed due to lack of memory. I > downloaded SQLite ver.3.6.23.1. I tried to use Firefox Data Manager add-on > but it would not load the csv files - 'csv worker failed'. So I tried > Database Master from Nucleon but it failed after loading (it took 100 > minutes) ~57,000 rows with error message = 'database or disk is full". I > tried to create another table in the same db but could not with same error > message. The DB size shows as 10,000KB (that looks suspiciously like a size > setting?). > > > > From what I've read SQLite can handle this size DB. So it seems that either > I do not have enough RAM or there are memory/storage (default) limits or > maybe time-out issues that prevent loading this large file . or the 2 GUI > tools I tried have size limits. I do have a fast server (16GB, 12 procs, > 64-bit intel, Win server) and an iMAC available. > > > > 1. Is SQLite the wrong tool for this project? (I don't want the > overkill and admin overhead of a large MySQL or SQL Server, etc.) > > 2. If SQLite will work, are there configuration settings in SQLite or > Win7 that will permit the load . or is there a better tool for this > project? > > > > Thanks much for helping a newbie! > > > > peterK > > > > _______________________________________________ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > _______________________________________________ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > -- Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the Universe trying to produce bigger and better idiots. So far, the Universe is winning. - Rich Cook _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users