Re: [appengine-java] Massive delete (mapreduce based) Datastore viewer Datastore statistics unavailable

2010-12-10 Thread Stephen Johnson
Are you still having this issue? I'm curious to learn if this has cleared up
because here's a random guess. When entities are deleted they are just
marked as deleted and then later garbage collected, so perhaps the datastore
viewer has a very low timeout set and it is starting with the beginning of
the key index and having to scan millions of those deleted entities and is
timing out before it finds enough results. Your own queries might be working
because they are using a different index than the key index and have the
full default timeout period and are finding the entities to satisfy your
queries. Has this magically cleared up for you. If so, it might be because
the garbage collection mechanism has removed those deleted entities. Anyway
it is just a guess and I'm curious if this is the case because it might be
that people may want to run slower delete jobs so as not to impact their
performance.

On Thu, Dec 9, 2010 at 2:07 AM, Cyrille Vincey crll...@gmail.com wrote:

 My problem:
 1. I execute a massive delete over a large number of entities (30 million),
 using a mapreduce-based job similar to Ikai's one (*)
 2. Then both the datastore viewer and the datastore statistics in the
 dashboard become unavailable: I get this famous the server encountered an
 error (blahblah) page.

 Yet, servlet-based queries against the datastore are ok, and the datastore
 indexes viewer do not show any error.
 Any clue ?

 (*)
 http://ikaisays.com/2010/07/09/using-the-java-mapper-framework-for-app-engine
 /

  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine for Java group.
 To post to this group, send email to
 google-appengine-j...@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2bunsubscr...@googlegroups.com
 .
 For more options, visit this group at
 http://groups.google.com/group/google-appengine-java?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine for Java group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.



[appengine-java] Massive delete (mapreduce based) Datastore viewer Datastore statistics unavailable

2010-12-09 Thread Cyrille Vincey
My problem:
1. I execute a massive delete over a large number of entities (30 million),
using a mapreduce-based job similar to Ikai's one (*)
2. Then both the datastore viewer and the datastore statistics in the
dashboard become unavailable: I get this famous the server encountered an
error (blahblah) page.

Yet, servlet-based queries against the datastore are ok, and the datastore
indexes viewer do not show any error.
Any clue ?

(*) 
http://ikaisays.com/2010/07/09/using-the-java-mapper-framework-for-app-engin
e/



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine for Java group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.