On 17 Sep, 23:25, djerdo <grflana...@gmail.com> wrote:
> Using the bulkloader with the dev appserver, the script adds rows
> (Entities) at a progressively slower rate, to the point where it
> becomes unusable when the csv file is large (20,000 rows). Why? Is
> this a known issue? Are there any workarounds?
>
> Thanks
Ok, I wasn't doing it right. FWIW - to resolve reference property
fields, I was using code such as:
select __key__ from MyModel where id = :1
id being the unique field in the incoming csv file. This seemed to
work but when i figured out how to specify a key_name when adding an
entity, I could replace this with
MyModel.get_by_key_name(id).key()
which sped things up a lot.
How to specify a key name when bulkloading entities? I couldn't find
any examples of this, so (again FWIW), here's a 'Baseloader' class
that i used:
from google.appengine.api import datastore
from google.appengine.ext import bulkload
class BaseLoader(bulkload.Loader):
#abstract class
def __init__(self):
bulkload.Loader.__init__(self, self.model, self.mapping)
def HandleEntity(self, entity):
if not hasattr(self, 'unique_field'):
return entity
#entity is a dict subclass
keyname = entity.pop(self.unique_field)
named_entity = datastore.Entity(self.model, name=keyname)
named_entity.update(entity)
return named_entity
Example:
class ContactLoader(Loader):
model = 'Contact'
mapping = [
('id', str),
('name', str),
('phone', str),
]
unique_field = 'id'
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~----------~----~----~----~------~----~------~--~---
