Here's a better definition of the problem: I receive a *value* for a tracked metric (i.e. start-up time) for different systems at a rate of 15/s. This rate is expected to grow quickly as clients add systems. I need to produce a rolling median of that metric.
Inserting all entries in the datastore is not an option since that would already require 15 writes per second which is extremely expensive. Using the memcache is not a good solution since there is no atomic push/pop on arrays. (Hence, my earlier reference to Redis.) Using a backend instance to hold it all in memory is a quick fix but it won't scale as we add new metrics. At the same time, I'm trying to keep the cost low since volume is only going to grow. On Wednesday, November 13, 2013 2:20:21 PM UTC-5, Kaan Soral wrote: > > A single datastore entity can hold up to 1MB's > > How big will a single dataset be? > > If it's smaller than 1MB's in summarized format, you could build a queue > based solution to handle the >15/s data rate > > You could also probably develop something like a tree, with each entity > representing a node and storing the data about the leafs etc, it could > maybe lead up to a practical median calculator, just an idea, the point is: > > as Vinny P stated, the solution is always based on exactly what you are > doing > > On Tuesday, November 12, 2013 10:07:34 PM UTC+2, Mathieu Simard wrote: >> >> Since there is no appengine solution available such as the Redis atomic >> list, I'm left wondering how to implement a cost effective rolling median. >> Has anyone come up with a solution that would be more convenient than >> running a redis instance on Google Compute Engine? >> > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
