Jeff - I'm a bit confused. I thought that the whole idea of the datastore was that you could read or write as much as you want, as fast as you want as long as they are not related? So one datastore write per vote (and being written to different entity groups) should be fine? I thought that the system just split tablets if they were being accessed too much - so as long as the traffic didn't suddenly increase, there'd be no scalability issues apart from cost.
Have I misunderstood? Thanks, Mat. On Sep 30, 9:15 pm, Jeff Schnitzer <[email protected]> wrote: > Assuming the goal is massive write throughput, you don't want to do 1 > write per vote. You need to aggregate writes into a batch - you can > do that with pull queues, but then you're limited to the maximum > throughput of a pull queue. And the biggest batch size is 1,000 > which might actually be votes for 1,000 different things, which means > you're back to 1-vote-1-write. > > Peter, you can certainly build a system whereby all vote totals are > tracked in RAM in a backend but now you're putting higher memory > requirements on the backend and life gets more complicated when you > deal with sharding. > > Depending on how exact you need the counts to be, you can always use > increment() on memcache in addition to incrementing the backend. The > only catch is that bootstrapping the initial memcache value will be a > little tricky - you'll need to use CAS and query the backends for any > existing deltas. Or just not care if it's off by a few. > > Jeff > > > > > > > > On Thu, Sep 29, 2011 at 5:56 AM, Mat Jaggard <[email protected]> wrote: > > Will that end up being cheaper than the following? > > > Put a new entity in the datastore for each vote - Kind: "Vote", ID: > > Auto generated, Vote for: "Item S" > > Have a task queue query all the votes and delete them then write the > > count of votes to a global object. > > > Cost = 1 datastore read + 1 datastore write + some fairly minor task > > processing per vote. > > > On Sep 29, 1:47 pm, Peter Dev <[email protected]> wrote: > >> Price: > >> - with backends lets say 3 B2 machines = 350USD/Month > >> - UrlFetch Data Sent/Received 0,15USD/GB > > >> Limit: > >> - URL Fetch Daily Limit 46,000,000 calls > >> this can be a problem...but I see it is possible to request an > >> increase > > >> Write data parallel in DB: Task Queue with rate every 30second could > >> be a solution > >> (check timestamps in cache and write in DB) > > >> RESET counters = empty cache in Backends & reset counter of object in > >> DB > > >> Backends cache = HashMap with shared counter values > >> or > >> counter values without sharding > >> (just increment value in java hashmap is fast enough) > > >> With backends we don’t need sharding I think....what do you think? Thx. > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine for Java" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]. > > For more options, visit this group > > athttp://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
