Re: [sqlite] Boosting insert and indexing performance for 10 billion rows (?)

Thank you for the good suggestions. I've been applying them to a smaller subset of my database to see how it might perform. I had tried fiddling with the cache size but it seemed to make performance slightly degrade in my case. In principle it should work, so perhaps my smaller database isn't enti

Re: [sqlite] Boosting insert and indexing performance for 10 billion rows (?)

Simon, Dominique, and Arun - Thank you for the feedback. I'll leave the threading off for inserts since I've seen DB contention issues with other multithreaded/multiprocessed attempts. The indexing improvement is nice though. I misspoke when I said it was 200-300gb for just the integers -- my bra

Re: [sqlite] Boosting insert and indexing performance for 10 billion rows (?)

Ah, this tool seems very handy. For those curious, I'll paste the results below. The index approximately doubles the storage size, but I am intentionally making that tradeoff to avoid the slow down when enforcing a unique/primary key on the Reference table while inserting. -AJ /** Disk-Space Util