Re: [web2py] Best way to insert 200k records?

[Vasile Ermicioi]

I would do commit after each 1 inserts


data #some array of dicts
for i in xrange(len(data)):
db.table.insert(**data[i])
if i%1==0:
db.commit()
db.commit()

-- 

Re: [web2py] Best way to insert 200k records?

[vinicius...@gmail.com]

I'd choose bulk_insert, but you should run some benchmarks, too.


On 08/17/2012 09:08 AM, Mike Girard wrote:

But what method should I use? Is there anything better than bulk_insert? 
Naturally I would like this to work as fast as possible.

Thanks.



--

Re: [web2py] Best way to insert 200k records?

[Mike Girard]

But what method should I use? Is there anything better than bulk_insert? 
Naturally I would like this to work as fast as possible.

Thanks. 

-- 

Re: [web2py] Best way to insert 200k records?

[vinicius...@gmail.com]

Maybe you can face transaction size problems with this volume.

Web2py, by default, just commits after a successful transaction. So, try 
to db.commit() after some transaction completion identification.


But I think processing time wouldn't be your problem.

--
Vinicius Assef



On 08/17/2012 08:50 AM, Mike Girard wrote:

Hello:

What is the prescribed method for doing large bulk inserts? I am using sqllite 
as my backend.

Is it ok to just write a Python script that talks to the db directly or should 
the DAL be used? The book says the bulk_insert method is not more advantageous 
than a for loop.

I searched this group for an answer but didn't find anything definitive.

Thanks.

PS Kudos to the web2py creator and contributors. Every day I am struck by how 
elegant, easy and well-designed it is.



--