I'm new to Python and GAE too, so bear with me if I'm saying something stupid. I read about how backups of whole GEA databases were made and the technique involves using both ordering and filtering to achieve an iterator on a large list.
I principle it goes like this, you order the list, let's say in your case by datetime, than you read the first thousand records. Then after to "requery" the the store, but by filtering for dates greater than (or lower than) the last date your processed. I'm not sure how deterministic the store sort is for entries with the same value (i.e. datetime) and if duplicate are possible for you application. Maybe ordering by key value would be more deterministic, since they are unique. Sebastien On Aug 12, 5:22 am, "Nick Johnson (Google)" <nick.john...@google.com> wrote: > Hi Adam, > > > > > > On Tue, Aug 11, 2009 at 5:12 AM, ajacks504<adam.p.jack...@gmail.com> wrote: > > > Hi All, > > > I'm really new to python and databases period, so please take it easy > > on me, im trying to learn. im trying to roll my own energy monitor > > and i cant really figure out how to get the number of entities in my > > data store. > > > class PowerData(db.Model): > > date = db.DateTimeProperty(auto_now_add=True) # timestamp > > kw = db.FloatProperty() > > # current kilowatt data (0.01 scale) > > > my database will have more than 1000 entires in it quickly. i want to > > grab 1000 data points and send it to the google annotated timeline > > display, but i know my query will be limited to 1000 results. I want > > to write a query that will return the number of entities in my > > datastore, then write a function that decimates (get every Nth one) > > the data based on a hop factor that will give me less than 100 data > > points. > > > 1- i have no idea how to get the full size of the datastore if its > > greater than 1000 entities > > There's no efficient way to do this, simply because there's no way to > count something without spending O(n) time doing it. You need to > maintain a count of entities in another entity, and update it when you > add new records. > > > 2- i dont know how to grab the decimated data in a way that wont be > > computationaly expensive > > You could add a property that you set randomly to a number between 0 > and 9. Fetching every record where that property equals (for example) > 0 will get you 1/10 the records. > > Alternately, you can have a task that 'rolls up' every 10 entities > into one aggregate record, using the task queue or otherwise. > > > > > im not looking for handout code, just throw me a bone for something to > > read up on? > > > is this a decent way to structure the data base? should i change it > > before i got further? > > > thanks, > > adam > > -- > Nick Johnson, Developer Programs Engineer, App Engine --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---
