Re: [sqlite] Bulk Insert in Sqlite3

On Feb 6, 2017, at 10:36 PM, Niti Agarwal wrote: > I read about SQLITE_MAX_SQL_LENGTH, If this is why you’re making many transactions, there’s no requirement that all of the SQL that’s part of a single transaction be in a single SQL string given to the DB. You can

Re: [sqlite] Bulk Insert in Sqlite3

On Tue, 7 Feb 2017 14:42:13 +0800 Rowan Worth wrote: > Note that golang's sql transaction abstraction doesn't map perfectly > to sqlite. Golang does not allow any further operations on the Tx > following a call to Tx.Commit() or Tx.Rollback(). But in sqlite a > transaction

Re: [sqlite] Bulk Insert in Sqlite3

On 7 February 2017 at 15:11, Simon Slavin wrote: > > On 7 Feb 2017, at 6:56am, Niti Agarwal wrote: > > > Thanks for your reply. The length matters as I am appending 100 rows at a > > time in a sql statement. It is making very fast as compared to

Re: [sqlite] Bulk Insert in Sqlite3

On 7 Feb 2017, at 6:56am, Niti Agarwal wrote: > Thanks for your reply. The length matters as I am appending 100 rows at a > time in a sql statement. It is making very fast as compared to single sql > insert in For loop. > Copied the code below for reference. Here the

Re: [sqlite] Bulk Insert in Sqlite3

Thanks for your reply. The length matters as I am appending 100 rows at a time in a sql statement. It is making very fast as compared to single sql insert in For loop. Copied the code below for reference. Here the list size is 100 Any better way to do this? Like I read about *bind*...not sure how

Re: [sqlite] Bulk Insert in Sqlite3

Hi Niti, There's on need to build a giant SQL string; a transaction can span multiple statements. To bind in golang place a ? within your SQL query and provide the values as additional arguments to the Exec/Query function. eg, after using db.Begin() to create a transaction tx, err :=

Re: [sqlite] Bulk Insert in Sqlite3

On 7 Feb 2017, at 5:36am, Niti Agarwal wrote: > Need to insert close to 10 Million records to sqlite3 in around 30 mins. This number of records requires so much space the temporary data will not fit inside a cache. Consider using a counter so that the transaction is

[sqlite] Bulk Insert in Sqlite3

Hi, We are using Sqlite3 with Golang to do bulk insert. Need to insert close to 10 Million records to sqlite3 in around 30 mins. Currently I am saving 100 Records under one transaction with below settings: PRAGMA synchronous = NORMAL; PRAGMA journal_mode = WAL; PRAGMA