On Sun, Sep 27, 2009 at 1:07 PM, djerdo <grflana...@gmail.com> wrote:
>
> Hi Nick,
>
> Thanks, that's more straightforward. Somewhat related, I'm creating an
> auto-generated 'bulk-delete' script that uses the 'remote_api'
> handler. Since i know the key_name for each entity that I'm uploading,
> I create a script which looks something like (ignoring imports):
>
> ---- deleteall_mymodel.py ------
> def auth_func():
> return raw_input('Username:'), getpass.getpass('Password:')
>
> app_id = 'myapp'
> host = app_id + '.appspot.com'
>
> remote_api_stub.ConfigureRemoteDatastore(app_id, '/remote_api',
> auth_func, host)
>
> print("Deleting entity: a1")
> obj = MyModel.get_by_key_name("a1")
> if obj is not None:
> obj.delete()
> print("Deleting entity: a2")
> obj = MyModel.get_by_key_name("a2")
> if obj is not None:
> obj.delete()
> print("Deleting entity: a3")
> obj = MyModel.get_by_key_name("a3")
> if obj is not None:
> obj.delete()
> ......
>
> -----------------------------------------------------
>
> This creates a huge file but it does work, and at a not unreasonable
> speed. What do think of this approach?
>
You could do this much more efficiently by loading the list of keys into a
list (say, from a text file), and deleting them in batches, something like
this:
f = iter(open("todelete.txt", "r"))
for batch in zip(*([f]*20)):
db.delete(batch)
Note there's also no need to fetch the entity before deleting it.
-Nick Johnson
> Thanks
>
> G.
>
> On Sep 26, 8:17 pm, "Nick Johnson (Google)" <nick.john...@google.com>
> wrote:
> > Hi djerdo,
> >
> > You can specify key names by overriding the generate_key method of your
> > loader. See here for more details:
> http://blog.notdot.net/2009/9/Advanced-Bulk-Loading-part-2-Customization
> >
> > -Nick Johnson
> >
> >
> >
> > On Sat, Sep 26, 2009 at 1:54 PM, djerdo <grflana...@gmail.com> wrote:
> >
> > > On 17 Sep, 23:25, djerdo <grflana...@gmail.com> wrote:
> > > > Using the bulkloader with the dev appserver, the script adds rows
> > > > (Entities) at a progressively slower rate, to the point where it
> > > > becomes unusable when the csv file is large (20,000 rows). Why? Is
> > > > this a known issue? Are there any workarounds?
> >
> > > > Thanks
> >
> > > Ok, I wasn't doing it right. FWIW - to resolve reference property
> > > fields, I was using code such as:
> >
> > > select __key__ from MyModel where id = :1
> >
> > > id being the unique field in the incoming csv file. This seemed to
> > > work but when i figured out how to specify a key_name when adding an
> > > entity, I could replace this with
> >
> > > MyModel.get_by_key_name(id).key()
> >
> > > which sped things up a lot.
> >
> > > How to specify a key name when bulkloading entities? I couldn't find
> > > any examples of this, so (again FWIW), here's a 'Baseloader' class
> > > that i used:
> >
> > > from google.appengine.api import datastore
> > > from google.appengine.ext import bulkload
> >
> > > class BaseLoader(bulkload.Loader):
> > > #abstract class
> >
> > > def __init__(self):
> > > bulkload.Loader.__init__(self, self.model, self.mapping)
> >
> > > def HandleEntity(self, entity):
> > > if not hasattr(self, 'unique_field'):
> > > return entity
> > > #entity is a dict subclass
> > > keyname = entity.pop(self.unique_field)
> > > named_entity = datastore.Entity(self.model, name=keyname)
> > > named_entity.update(entity)
> > > return named_entity
> >
> > > Example:
> >
> > > class ContactLoader(Loader):
> > > model = 'Contact'
> > > mapping = [
> > > ('id', str),
> > > ('name', str),
> > > ('phone', str),
> > > ]
> > > unique_field = 'id'
> >
> > --
> > Nick Johnson, Developer Programs Engineer, App Engine
> > Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration
> Number:
> > 368047
> >
>
--
Nick Johnson, Developer Programs Engineer, App Engine
Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number:
368047
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~----------~----~----~----~------~----~------~--~---
