On Fri, Aug 5, 2016 at 3:03 PM, Darren Duncan <dar...@darrenduncan.net> wrote:
> On 2016-08-04 7:27 AM, Jim Callahan wrote: > >> Steps >> Agree with Darren Duncan and Dr. Hipp you may want to have at least 3 >> separate steps >> (each step should be a separate transaction): >> >> 1. Simple load >> 2. Create additional column >> 3. Create index >> >> Have you pre-defined the table you are loading data into? (step 0 CREATE >> TABLE) >> >> If "Step 1 Simple Load" does not complete; then may want to load a fixed >> number of rows into separate tables (per Darren Duncan) and then combine >> using an APPEND >> or a UNION query (doing so before steps 2 and 3). >> > > To be clear, my proposal of only loading a subset of rows was just > intended for debugging the performance issues the OP was having and try and > figure out what is causing the problem without having to wait too long for > larger sets to complete. I wasn't proposing splitting the load into > separate databases and unioning later, rather the attempt database would be > thrown away after each trial. -- Darren Duncan > Thanks, I understood that and it was a good set of suggestions. Using separate transactions led to focusing on the CREATE INDEX part of the process. That's where it gets wedged. Working up to the full database, by first processing smaller extracts, convinced me that size matters. Smaller (even millions of row) extracts do not wedge. -- #define QUESTION ((bb) || (!bb)) /* Shakespeare */ _______________________________________________ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
