Can use low-level datastore and iterate over batches:
binSize = 1000;// or smaller if needed < 30 sec timeout
DatastoreService datastore =
DatastoreServiceFactory.getDatastoreService();
Keys only select and delete:
javax.jdo.Query q = pm.newQuery("select key from " +
entityKind.getClass().getName());
q.setRange(0, binSize);
(List<Key>) q.execute();
ds.delete(keys);
Or slower entity-by-entity delete:
FetchOptions fetchOptions =
FetchOptions.Builder.withLimit(binSize);
com.google.appengine.api.datastore.Query q = new
com.google.appengine.api.datastore.Query(entityKind);
PreparedQuery pq = datastore.prepare(q);
List<com.google.appengine.api.datastore.Entity> results =
pq.asList(fetchOptions);
for (com.google.appengine.api.datastore.Entity result : results) {
com.google.appengine.api.datastore.Key key = result.getKey();
datastore.delete(key);
}
On Sep 9, 9:55 pm, Didier Durand <[email protected]> wrote:
> If you don't want to introduce MapReduce solely for this purpose, you
> can write a queued task that loops through the data to delete it ....
> and that recreates a new instance of itself just before time limit (10
> min) to continue the deletion.
>
> regards
>
> didier
>
> On Sep 9, 8:37 pm, Marcelo Liberato <[email protected]> wrote:
>
>
>
>
>
>
>
> > You may use MapReduce programmatically or just use Datastore admin (which
> > uses mapreduce behind the scenes).
>
> > On Thu, Sep 8, 2011 at 12:30 PM, blitzer <[email protected]> wrote:
> > > So I have ~ 35 million rows of data. I have seen that I need to get the
> > > data into a collection and then call deletePersistAll(). Is that the only
> > > way?
>
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "Google App Engine for Java" group.
> > > To view this discussion on the web visit
> > >https://groups.google.com/d/msg/google-appengine-java/-/gg8buxlS2JUJ.
> > > To post to this group, send email to
> > > [email protected].
> > > To unsubscribe from this group, send email to
> > > [email protected].
> > > For more options, visit this group at
> > >http://groups.google.com/group/google-appengine-java?hl=en.
--
You received this message because you are subscribed to the Google Groups
"Google App Engine for Java" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
