Yejun, I've been told that the memcached framework on GAE uses a first- created, first-deleted algorithm, and that we have finite (a few hundred megs, but maybe even up to a GB) of memcache for any specific app. This means that once you hit your limit, older objects WILL get deleted. And my app will definitely be going over this max limit. This is not a huge deal to me (probably won't happen that my counter gets deleted that often, and it's ok if it's a little bit off) but I figured my counter might want as well handle that small case anyway.
Again, the much bigger case: Not writing to the datastore each time you want to increment. And yes, I am aware of why to use the sharded counter. The point is what if you have about 50QPS coming in (so you need a sharded counter with a lot of shards for sure), and every single request is writing to ~3 different counters. Now each request is taking a while because of the datastore writes that it attempts each time, even with no Transaction collisions on shard-writes. And also I believe there is a GAE watchdog looking to see if over a period of time your average request is >300ms. So I am simply saying, why not try cut down on the total datastore writes, and write to a shard only 1/10 times, but still get the correct totals? This is the reasoning for my arguments above. So now you have an order of magnitude less datastore writes, and the average response time is way down. This sounds good to me, and I am sure others who plan to write apps that have a number of sharded counter increments / avg. request, might feel similarly. Am I missing something obvious here? -Josh On Nov 3, 12:54 pm, yejun <[EMAIL PROTECTED]> wrote: > I believe mecached algorithm will keep most used object not just by > creation time. > The reason you use a sharded count is that you want increment > operation always hit the disk. Otherwise you don't need shard because > you don't access it very often if write is cached in memcache. Shard > is used here is to improve write concurrency. > > On Nov 3, 3:43 pm, josh l <[EMAIL PROTECTED]> wrote: > > > Yejun, > > > Thanks for the updated code example. Since a counter is such a common > > need, I think it might be helpful if we all worked together on the > > same codebase, rather than forking each time, or at least if we could > > be specific about the changes we made (I know I can do a diff, but if > > you noted what exactly you changed, and why, that would be awesome for > > all future users who are curious). > > > Moving on, I think I didn't explain myself well regarding the > > destruction of the memcache object. Imagine an app where > > 1) There will definitely be at least 10 counter requests/sec (for > > same named counter. let's call it the TotalRequests counter, and is > > referred to by some Stats model) > > 2) Lots and lots of other entities get written to memcache (millions > > of entities in the system, and each gets cached upon initial request) > > > In this situation, it is guaranteed objects in our memcache will > > disappear after some use, since we have less memcache total avail than > > the size of items that will be cached over a few days of use. Now, > > which items get removed? In this case, our counter is the first > > created item in memcache, and definitely one of the first items to be > > just nuked from memcache when we hit the max storage limit for our > > apps' memcache. To ensure it never gets nuked due to it being the > > 'oldest object in memcache', then we could 'occasionally' destroy/ > > recreate it. Maybe, for example, I could also have a time_created on > > it, and if it's older than a few hours, then nuke/recreate upon > > resetting it. I figured might as well do this every time, but anyway > > hopefully you see my point as to why I was thinking about the need to > > destroy/reuse. > > > Much more important than this very occasional mis-count for a > > destroyed memcache item, tho, is my general idea of just not even > > attempting to write to a shard entity unless we've had a few (10?, > > 50?) counter increments. I am getting ~350ms/request average due to > > the time it takes writing to the shards (multiple counters/request), > > and this is my main concern with the current code. > > > I will diff your code (thanks again) and check it out this afternoon. > > > -Josh > > > On Nov 3, 12:22 pm, yejun <[EMAIL PROTECTED]> wrote: > > > > > To solve this, I'm also > > > > planning to destroy and recreate the memcache object upon successful > > > > datastore write (and associated memcache delay being reset to zero). > > > > You shouldn't do this. It the completely negates the reason why you > > > use this counter class. > > > If you just need a counter for a local object, you should just save > > > the counter with object itself. > > > > This counter class should only be used in situation when the same > > > named counter need to be increment more than 10 times per second by > > > different requests/users concurrently. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---
