Have a look at nimbits it stores time series in appengine datastore. It's written in java, but the data models used should be straightforward to translate into NDB.
T On Friday, January 30, 2015 at 4:32:52 AM UTC+8, Shailendra Singh wrote: > > Hi Rafael > > It's a old thread, but can you please share some information on how you > stored " different rows for hour, day, week, month, year, etc. You can > squeeze a lot of data in 1mb :)" in GAE? I an new to GAE and i am trying to > store some ts data with respect to a entity in NDB. > > Thanks > > On Wednesday, August 14, 2013 at 4:50:15 AM UTC+5:30, Rafael Sanches wrote: >> >> i implemented this by having these components: >> >> - TimeSeriesIndex - different rows for hour, day, week, month, year, etc. >> You can squeeze a lot of data in 1mb :) >> - DataPoint - unprocessed data point data. thousands of rows per minute. >> - cron that process the datapoints inside the indexes >> - the ui read only TimeSeriesIndex that contains the timestamps and the >> data points. >> >> thanks >> rafa >> >> >> On Tue, Aug 13, 2013 at 1:42 PM, Jay <jbake...@gmail.com> wrote: >> >>> In my opinion, your biggest take away from this should be to avoid >>> having a mega entity group and you do this by simply *not* having all >>> the entities in question have the same parent. Or perhaps more pointedly, >>> any parent at all. Unless there is a really strong case to put many >>> thousands of entities in the same entity group, I just wouldn't do it. You >>> can have transactions across entity groups now so if you need a transaction >>> with a few entities you are OK. >>> As you need to relate the entities, do that by some other means instead >>> of a parent entity. For example, you could use a ndb.KeyProperty or >>> possibly just an encoded string or something along those lines. >>> >>> >>> On Tuesday, August 13, 2013 7:59:52 AM UTC-5, Martin Trummer wrote: >>>> >>>> I'm a newbie to the AppEngine datastore and like to know how to best >>>> design this use case: >>>> there may be some time-series with huge amount of data: e.g. >>>> terra-bytes for one time-series >>>> the transacations doc >>>> <https://developers.google.com/appengine/docs/java/datastore/transactions> >>>> says about entity groups: >>>> >>>> - *"Every entity belongs to an entity group, a set of one or more >>>> entities that can be manipulated in a single transaction."* >>>> - *"every entity with a given root entity as an ancestor is in the >>>> same entity group. All entities in a group are stored in the same >>>> Datastore >>>> node."* >>>> >>>> so does that mean, that all the terra-bytes of data for the huge >>>> time-series would end up *on one computer* somewhere in the AppEngine >>>> network? >>>> if so: >>>> >>>> - that's not a good idea, right? >>>> - how to avoid it? should I split up the data in sections (e.g. per >>>> month) where each section has it's own kind/entity group? >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Google App Engine" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to google-appengi...@googlegroups.com. >>> To post to this group, send email to google-a...@googlegroups.com. >>> Visit this group at http://groups.google.com/group/google-appengine. >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >>> >>> >> >> -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/51bfd5c0-adca-4770-8e69-1b7ae637f04b%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
