Hi Nick, Thanks, that's more straightforward. Somewhat related, I'm creating an auto-generated 'bulk-delete' script that uses the 'remote_api' handler. Since i know the key_name for each entity that I'm uploading, I create a script which looks something like (ignoring imports):
---- deleteall_mymodel.py ------ def auth_func(): return raw_input('Username:'), getpass.getpass('Password:') app_id = 'myapp' host = app_id + '.appspot.com' remote_api_stub.ConfigureRemoteDatastore(app_id, '/remote_api', auth_func, host) print("Deleting entity: a1") obj = MyModel.get_by_key_name("a1") if obj is not None: obj.delete() print("Deleting entity: a2") obj = MyModel.get_by_key_name("a2") if obj is not None: obj.delete() print("Deleting entity: a3") obj = MyModel.get_by_key_name("a3") if obj is not None: obj.delete() ...... ----------------------------------------------------- This creates a huge file but it does work, and at a not unreasonable speed. What do think of this approach? Thanks G. On Sep 26, 8:17 pm, "Nick Johnson (Google)" <nick.john...@google.com> wrote: > Hi djerdo, > > You can specify key names by overriding the generate_key method of your > loader. See here for more > details:http://blog.notdot.net/2009/9/Advanced-Bulk-Loading-part-2-Customization > > -Nick Johnson > > > > On Sat, Sep 26, 2009 at 1:54 PM, djerdo <grflana...@gmail.com> wrote: > > > On 17 Sep, 23:25, djerdo <grflana...@gmail.com> wrote: > > > Using the bulkloader with the dev appserver, the script adds rows > > > (Entities) at a progressively slower rate, to the point where it > > > becomes unusable when the csv file is large (20,000 rows). Why? Is > > > this a known issue? Are there any workarounds? > > > > Thanks > > > Ok, I wasn't doing it right. FWIW - to resolve reference property > > fields, I was using code such as: > > > select __key__ from MyModel where id = :1 > > > id being the unique field in the incoming csv file. This seemed to > > work but when i figured out how to specify a key_name when adding an > > entity, I could replace this with > > > MyModel.get_by_key_name(id).key() > > > which sped things up a lot. > > > How to specify a key name when bulkloading entities? I couldn't find > > any examples of this, so (again FWIW), here's a 'Baseloader' class > > that i used: > > > from google.appengine.api import datastore > > from google.appengine.ext import bulkload > > > class BaseLoader(bulkload.Loader): > > #abstract class > > > def __init__(self): > > bulkload.Loader.__init__(self, self.model, self.mapping) > > > def HandleEntity(self, entity): > > if not hasattr(self, 'unique_field'): > > return entity > > #entity is a dict subclass > > keyname = entity.pop(self.unique_field) > > named_entity = datastore.Entity(self.model, name=keyname) > > named_entity.update(entity) > > return named_entity > > > Example: > > > class ContactLoader(Loader): > > model = 'Contact' > > mapping = [ > > ('id', str), > > ('name', str), > > ('phone', str), > > ] > > unique_field = 'id' > > -- > Nick Johnson, Developer Programs Engineer, App Engine > Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: > 368047 --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---