Hi, Sorry for the repeat, but I am trying to wrap my head around the GAE-osphere and I am getting a bit confused;
I need to store and retrieve/analyse timeseries data, of varying sizes and resolutions; at the moment, the data is received and stored on GAE through to Google Cloud SQL (python). That's not ideal. I foresee I will have to do more analytics than data storage for the sake of storage, and I predict a big throughput of data generally, and have thus been looking at BigQuery/Datalab. I don't seem to find an obvious way to load data in BigQuery from Cloud SQL, and would either have to export the data to Cloud Storage in csv-ish, or directly stream the data from the GAE app to BigQuery (which is currently my preferred option). Alternatively, there is the option of passing through to Google Datastore first, which for me might be a more flexible way of preprocessing the data before it enters in BigQuery. Is this the way to do things, or am I missing something? Thanks! Etienne On Tuesday, 13 August 2013 13:59:52 UTC+1, Martin Trummer wrote: > > I'm a newbie to the AppEngine datastore and like to know how to best > design this use case: > there may be some time-series with huge amount of data: e.g. terra-bytes > for one time-series > the transacations doc > <https://developers.google.com/appengine/docs/java/datastore/transactions> > says about entity groups: > > - *"Every entity belongs to an entity group, a set of one or more > entities that can be manipulated in a single transaction."* > - *"every entity with a given root entity as an ancestor is in the > same entity group. All entities in a group are stored in the same > Datastore > node."* > > so does that mean, that all the terra-bytes of data for the huge > time-series would end up *on one computer* somewhere in the AppEngine > network? > if so: > > - that's not a good idea, right? > - how to avoid it? should I split up the data in sections (e.g. per > month) where each section has it's own kind/entity group? > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/3bfaf3d1-8f39-4fbd-a30f-2577b8428269%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
