Storing the data points isn't an option. I'm already receiving far too many data points to start writing them all. Unless they dramatically lower the write costs...
As for the in-memory approach, can you provide a scale at which you're using this technique? Do you have to ensure that the backend runs on a single thread? On Wednesday, November 13, 2013 6:59:49 AM UTC-5, Gilberto Torrezan Filho wrote: > > I miss some Redis functionality in App Engine as well. Memcache is just an > unreliable cache to hold some data for while... nothing more. > > To make such calculations which iterate over large sets of data, I use > backends with in-memory processing: loading part of the data from datastore > into memory, spawn multiple threads (if applicable) and iterate over data. > Ugly, strange, error-prone and sometimes slow, but it works. > > A bomb-to-kill-an-ant solution would be using Google BigQuery. I don't > like like the idea, but depending on your problem it can solve it for you. > > You can try to use some MapReduce processing as well. But since I'm using > Java (a not so loved language in App Engine, see servlet 3.0 > discussion<http://code.google.com/p/googleappengine/issues/detail?id=3091>) > MapReduce (Mapper, actually<http://code.google.com/p/appengine-mapreduce/>) > is too experimental to put in production (after the Conversion and Files > API, I learned my lesson: never ever ever use an experimental API in App > Engine). > > Anyway, you have several options to try. I just recommend you to avoid > storing large datasets on Memcache, since it's just a cache and can wipe > your data at any time - invalidating your calculations. > > On Tuesday, November 12, 2013 6:07:34 PM UTC-2, Mathieu Simard wrote: >> >> Since there is no appengine solution available such as the Redis atomic >> list, I'm left wondering how to implement a cost effective rolling median. >> Has anyone come up with a solution that would be more convenient than >> running a redis instance on Google Compute Engine? >> > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
