[google-appengine] Re: What is a pattern for keeping track of current users in google app engine?
Hi, If reading 1000+ user records(say keys) in ONE task execution is problem then it doesnt matter if you use Memcache or data store, your solution is not gonna be scalable to update 1000+ user record(for last poll time) in ONE task execution. What i understood your problem is that you want to keep every user's last poll time updated against there user data. And as a solution you are running kind of Timer thread in the form of Task Queus and update all user's last poll time who accessed/polled in last 30/n seconds. I am specifying a little different approach depending on my understanding about your problem Using memcache Solution 1) You dont need to create 30 poll cache or n poll cache. Just create one Setkey object and put it in memcache and your polling request handler will keep adding user's id/key in this cache.Using Set/HashSet will make sure that you dont see to worry about merging or duplicacy. After 30/n seconds your task start executing it will take all the data(SetKey from cache and then remove everything from set in memcache, so that your polling request handler will find it empty and then they will start adding adding new user id/key in it. Now it may be possible that you got a set of 1 or more from cache and task execution may not be able to update all in 30 seconds, so you may want to create another sub task who will take left over user key and do the same thing again. Basically you will put the left over ids in memcache again with different id and pass that memcach id to sub-task and that task can read it from there. Or to make it more efficient, if you already know by experience that you are able to update on 500 records in one execution then divide your original left over user ids and divide it into set of 500 and create as many task as you want and save set of 500 in memcache with different ids like LEFT_OVER_SET_1, LEFT_OVER_SET_2 etc. and each task can read daat from memcache using these provided ids and if something left over it will try to invoke sub task again(divide by 500, but this time you will find total number is already less then 500). But still i say memcach edoesnt guarntee that over a span of time data will still be avaiable so instead of memcache you can use datastore in above solution. Ravi. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: What is a pattern for keeping track of current users in google app engine?
I'm just giving a not-well-thought suggestion, but I'd rather use memcache for this. I guess something like using a time-dependent code as the key (i.e. a counter which increments every 30 seconds) and a list of user ids as the value, and simply doing a get() on the cache for the current counter value would work. I don't know how this could scale, due to concurrent updates, but I definitely think Memcache is the way to go, not datastore. Regards Lorenzo On Jun 23, 5:45 am, Ryan ryeterr...@gmail.com wrote: Each of my users is polling the server every few seconds. I need to keep a list of the users that have polled in the last 30 seconds handy for a task I have queued to run every few seconds. The obvious way I see to do it is to update a datastore entry every time the user polls, and query the entries that have a timestamp within the last N seconds within my task queue. I can't imagine this scaling well. Any recommendations? Thanks. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: What is a pattern for keeping track of current users in google app engine?
I wouldnt suggest Memcache for this purpose as memcache doesnt guarntee that what you put in it will be available after 30 seconds. Memcach can be flushed because of number of reasons. And memcache defnitely will not be scalable, if your number of ysers goes high then least used entries will be deleted automatically even if that entry comes under criteria of last 30 seconds. I think your first thought about using datastore is good enough. Why do you think Datastore will not be scalable? May be you can explain your problem statement bit more. On Jun 23, 9:11 am, l.denardo lorenzo.dena...@gmail.com wrote: I'm just giving a not-well-thought suggestion, but I'd rather use memcache for this. I guess something like using a time-dependent code as the key (i.e. a counter which increments every 30 seconds) and a list of user ids as the value, and simply doing a get() on the cache for the current counter value would work. I don't know how this could scale, due to concurrent updates, but I definitely think Memcache is the way to go, not datastore. Regards Lorenzo On Jun 23, 5:45 am, Ryan ryeterr...@gmail.com wrote: Each of my users is polling the server every few seconds. I need to keep a list of the users that have polled in the last 30 seconds handy for a task I have queued to run every few seconds. The obvious way I see to do it is to update a datastore entry every time the user polls, and query the entries that have a timestamp within the last N seconds within my task queue. I can't imagine this scaling well. Any recommendations? Thanks.- Hide quoted text - - Show quoted text - -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: What is a pattern for keeping track of current users in google app engine?
On Jun 23, 2010, at 4:11 PM, l.denardo wrote: I'm just giving a not-well-thought suggestion, but I'd rather use memcache for this. I guess something like using a time-dependent code as the key (i.e. a counter which increments every 30 seconds) and a list of user ids as the value, and simply doing a get() on the cache for the current counter value would work. Alternatively, if you only want to see whether the user has polled, you could use the key 'user_%s_polled', substituting %s for the user's id, and place it in memcache with an expiration time of 30s. Every time you update the key, you can reset the expiration time. In Python, this would look like: memcache.set(user_%s_polled % user, True, time=30) Simply getting the key would tell you whether or not the user has polled in the last 30s. Memcache would then handle expiration; after 30s, the key is removed for you. Jan Michael Ibanez jmiba...@gmail.com -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: What is a pattern for keeping track of current users in google app engine?
I'm guessing it wont be scalable because if I have 1000+ polling users, I would have to query and fetch that many entries every time my task runs every second or so. Maybe that's not so bad? It seems like a lot of querying and fetching to me. On Wed, Jun 23, 2010 at 4:01 AM, ping2ravi ping2r...@gmail.com wrote: I wouldnt suggest Memcache for this purpose as memcache doesnt guarntee that what you put in it will be available after 30 seconds. Memcach can be flushed because of number of reasons. And memcache defnitely will not be scalable, if your number of ysers goes high then least used entries will be deleted automatically even if that entry comes under criteria of last 30 seconds. I think your first thought about using datastore is good enough. Why do you think Datastore will not be scalable? May be you can explain your problem statement bit more. On Jun 23, 9:11 am, l.denardo lorenzo.dena...@gmail.com wrote: I'm just giving a not-well-thought suggestion, but I'd rather use memcache for this. I guess something like using a time-dependent code as the key (i.e. a counter which increments every 30 seconds) and a list of user ids as the value, and simply doing a get() on the cache for the current counter value would work. I don't know how this could scale, due to concurrent updates, but I definitely think Memcache is the way to go, not datastore. Regards Lorenzo On Jun 23, 5:45 am, Ryan ryeterr...@gmail.com wrote: Each of my users is polling the server every few seconds. I need to keep a list of the users that have polled in the last 30 seconds handy for a task I have queued to run every few seconds. The obvious way I see to do it is to update a datastore entry every time the user polls, and query the entries that have a timestamp within the last N seconds within my task queue. I can't imagine this scaling well. Any recommendations? Thanks.- Hide quoted text - - Show quoted text - -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: What is a pattern for keeping track of current users in google app engine?
If I do it this way, how can I know which user id to try to get from the memcache? On Wed, Jun 23, 2010 at 4:14 AM, Jan Michael Ibanez jmiba...@gmail.comwrote: On Jun 23, 2010, at 4:11 PM, l.denardo wrote: I'm just giving a not-well-thought suggestion, but I'd rather use memcache for this. I guess something like using a time-dependent code as the key (i.e. a counter which increments every 30 seconds) and a list of user ids as the value, and simply doing a get() on the cache for the current counter value would work. Alternatively, if you only want to see whether the user has polled, you could use the key 'user_%s_polled', substituting %s for the user's id, and place it in memcache with an expiration time of 30s. Every time you update the key, you can reset the expiration time. In Python, this would look like: memcache.set(user_%s_polled % user, True, time=30) Simply getting the key would tell you whether or not the user has polled in the last 30s. Memcache would then handle expiration; after 30s, the key is removed for you. Jan Michael Ibanez jmiba...@gmail.com -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: What is a pattern for keeping track of current users in google app engine?
You wouldn't - the concept to use mcache would be to make a hash or a token for the time second'th element i.e 1-30 store each pole user in the correct element and then read the users back out of mcache by looping the 30 or mcache containers I did give this some thought last night when i saw your post but i ran out of time the idea would be lets say we use pole+1 to 30 as a key pole1 - happened 1 sec ago pole29 - happened 29 sec ago pole30 - will be removed when 31 secs elapse - as we set the mcache to destroy poles after 30 seconds then when somone poles we get the time convert it into a number 1-30 being second elements in a min - this need some thought maybe seconds devide by 2 - something like that we then add the user_key to to the pole. maybe a dictionary? then every second when you task runs you load using a loop pole1pole30 grab all the keys in the dict and make a list then if you need to load the models for the user you go a db.get(list) i think that starting to sound like it might work - sure some bright spark can add some more detail hope that helps Regards Martin Webb The information contained in this email is confidential and may contain proprietary information. It is meant solely for the intended recipient. Access to this email by anyone else is unauthorised. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted in reliance on this, is prohibited and may be unlawful. No liability or responsibility is accepted if information or data is, for whatever reason corrupted or does not reach its intended recipient. No warranty is given that this email is free of viruses. The views expressed in this email are, unless otherwise stated, those of the author From: Rye Terrell ryeterr...@ryeterrell.net To: google-appengine@googlegroups.com Sent: Wed, 23 June, 2010 19:23:53 Subject: Re: [google-appengine] Re: What is a pattern for keeping track of current users in google app engine? If I do it this way, how can I know which user id to try to get from the memcache? On Wed, Jun 23, 2010 at 4:14 AM, Jan Michael Ibanez jmiba...@gmail.com wrote: On Jun 23, 2010, at 4:11 PM, l.denardo wrote: I'm just giving a not-well-thought suggestion, but I'd rather use memcache for this. I guess something like using a time-dependent code as the key (i.e. a counter which increments every 30 seconds) and a list of user ids as the value, and simply doing a get() on the cache for the current counter value would work. Alternatively, if you only want to see whether the user has polled, you could use the key 'user_%s_polled', substituting %s for the user's id, and place it in memcache with an expiration time of 30s. Every time you update the key, you can reset the expiration time. In Python, this would look like: memcache.set(user_%s_polled % user, True, time=30) Simply getting the key would tell you whether or not the user has polled in the last 30s. Memcache would then handle expiration; after 30s, the key is removed for you. Jan Michael Ibanez jmiba...@gmail.com -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: What is a pattern for keeping track of current users in google app engine?
On 10-06-23 20:45, Martin Webb wrote: You wouldn't - the concept to use mcache would be to make a hash or a token for the time second'th element i.e 1-30 store each pole user in the correct element and then read the users back out of mcache by looping the 30 or mcache containers I did give this some thought last night when i saw your post but i ran out of time the idea would be lets say we use pole+1 to 30 as a key pole1 - happened 1 sec ago pole29 - happened 29 sec ago pole30 - will be removed when 31 secs elapse - as we set the mcache to destroy poles after 30 seconds then when somone poles we get the time convert it into a number 1-30 being second elements in a min - this need some thought maybe seconds devide by 2 - something like that Surely not, but seconds%30 should work. However, this way you store users who were polling multiple time during the last 30 seconds in multiple poles. When computing the user list, you need to read all 30 poles and merge them. Maybe saving a list of pairs (user, time) in a single pole could be better, and a dictionary mapping users to time would be even better. Each time you would set the value for the current user. From time to time you should evict too old entries, so the data doesn't grow too much. This way, you could use multiple poles too, let's say four of them, indexed by minutes%4 with a timeout like 1 minute. This prevents the data growth and computing the user list you need to look at only the last one or two poles. If there were thousands of users active in the last minute, the data could be so large, that the overhead with reading and storing might be high (no idea how efficient memcache loads and stores work). In such a case splitting users in groups (determined eg. by something like username.hashCode()%8 could help. I'm not sure, if this is the way to go, but it sounds quite simple. we then add the user_key to to the pole. maybe a dictionary? then every second when you task runs you load using a loop pole1pole30 grab all the keys in the dict and make a list then if you need to load the models for the user you go a db.get(list) i think that starting to sound like it might work - sure some bright spark can add some more detail hope that helps -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: What is a pattern for keeping track of current users in google app engine?
The key for the pole would be made as follows second = #get the min interval of the current time if second29 second=second-30 #this gives us 0-29 and then 0-29 again for one min then make the mcache key as key='pole'+second I don't know how many users may pole in one second - but i cant see merging a list stored in each pole would be too much of a handful. If 30 poles are too many use 15 - 1 for each 2 second element or 3 Further more i don't know what info you need on the user - if all you want is their identifier - that can be stored in the list or their email this would save on needing to get the users data from the model using the key. Martin You wouldn't - the concept to use mcache would be to make a hash or a token for the time second'th element i.e 1-30 store each pole user in the correct element and then read the users back out of mcache by looping the 30 or mcache containers I did give this some thought last night when i saw your post but i ran out of time the idea would be lets say we use pole+1 to 30 as a key pole1 - happened 1 sec ago pole29 - happened 29 sec ago pole30 - will be removed when 31 secs elapse - as we set the mcache to destroy poles after 30 seconds then when somone poles we get the time convert it into a number 1-30 being second elements in a min - this need some thought maybe seconds devide by 2 - something like that Surely not, but seconds%30 should work. However, this way you store users who were polling multiple time during the last 30 seconds in multiple poles. When computing the user list, you need to read all 30 poles and merge them. Maybe saving a list of pairs (user, time) in a single pole could be better, and a dictionary mapping users to time would be even better. Each time you would set the value for the current user. From time to time you should evict too old entries, so the data doesn't grow too much. This way, you could use multiple poles too, let's say four of them, indexed by minutes%4 with a timeout like 1 minute. This prevents the data growth and computing the user list you need to look at only the last one or two poles. If there were thousands of users active in the last minute, the data could be so large, that the overhead with reading and storing might be high (no idea how efficient memcache loads and stores work). In such a case splitting users in groups (determined eg. by something like username.hashCode()%8 could help. I'm not sure, if this is the way to go, but it sounds quite simple. we then add the user_key to to the pole. maybe a dictionary? then every second when you task runs you load using a loop pole1pole30 grab all the keys in the dict and make a list then if you need to load the models for the user you go a db.get(list) i think that starting to sound like it might work - sure some bright spark can add some more detail hope that helps -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: What is a pattern for keeping track of current users in google app engine?
I have more then 1000 users online at peak and I'm using datastore for this. Unfortunately, updating last online time costs quite a lot of CPU time. And I'm updating only once per 5 minutes (on each poll I write last online time to memcache and put it into datastore via cron). On Jun 23, 6:45 am, Ryan ryeterr...@gmail.com wrote: Each of my users is polling the server every few seconds. I need to keep a list of the users that have polled in the last 30 seconds handy for a task I have queued to run every few seconds. The obvious way I see to do it is to update a datastore entry every time the user polls, and query the entries that have a timestamp within the last N seconds within my task queue. I can't imagine this scaling well. Any recommendations? Thanks. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
