[google-appengine] What is a pattern for keeping track of current users in google app engine?
This is dragging a bit but its an interesting debate: By using 30 poles - what happens is as use pole in a nano second the pole n will be read and set with a 29 second removal span. So when the last nano second polling user hits that pole it sets it to remove in 30 seconds - so if ten users pole in that second they all simply re-set the removal for 30 seconds - and all we cause is a nano second gloat. If however you use two poles as you suggest if someone poles at 29 seconds the pole is set and it wont get removed for another 29 seconds - which basically means the first person to pole at 1 second will be kept for what is that 58 seconds? if you then think OK ill calculate a removal time based on how many seconds are left - ie if someone poles at 15 seconds when you set mcache you set 15 seconds remaning to remove not 30 then that persons pole is kept for 15 not 30 as i said in my last post. All of this sounds highly complex. As i said 30 poles solves all these worries and is simple to code and execute or am i missing the "rabit in the hat trick"? Interesting debate :-) This still will not work at if you use two sets they both expire within 30 secs so if i poll at 29 secs - in 1 sec the my poll is removed it should be kept for 29 secods - if i poll on say 45 seconds my poll is lost in 15. The idea is to show a clear window of "who" polled in the last the seconds. Or that's how i understand the requirement So if a poll come at 03:12:09(XX:YY:09) AM it will go to cache MY_USER_CACHE_FIRST - #this person must exist for 30secs not 21# and if request comes at 03:12:35(XX:YY:35) AM it will go to cache MY_USER_CACHE_LAST. #this person must exist for 30 secs not 25# the only way to keep a track to the nearest second is with 30 poles 15 poles gives you a two second error 10 poles gives you a three second error 5 poles gives you a 6 second error 2 poles gives you a 15 second error 30 pole solution is the only way to do it if you want to use mcache in this way. Further on the note of reading the users. I don't know what the original developers was planing on doing with the data - he mentioned he wanted a task to do "something" every few seconds and their maybe a lot of users. On that note assuming he is NOT updating the users model - assuming maybe its an admin page with an Ajax update to show who is polling then each time a user poles(which i know would not be a task) he should store a dictionary with the data from the user he needs in the pole - when they poll - not load it every n second in his task. Its better to read it once - but again this is all guess work. Finally if he is updating the users table it would be just as easy to update a polled time and run a task that draws which ones polled in the time frame and do the updateamd as you rightly say perhaps do 100 at a time and make the task re-run when their are poles left. The solution i posed was merely the approach that would work in mcache and was a simple one. I hope this is now agreeable? From: Ravi Sharma To: google-appengine@googlegroups.com Sent: Thu, 24 June, 2010 19:40:50 Subject: Re: [google-appengine] Re:What is a pattern for keeping track of current users in google app engine? Yes i understand about that there might be lost in few entries. That you can fix in this solution by many ways..one of the way by keeping two sets then.( i thought of explaning it before but mail was getting longer :) ) Ok let me explain in detail But this to work the both cache need to have seprate Id like and expire time say 30 second or more. KeyValue description MY_USER_CACHE_FIRST Set users for poll request between 00 to 30 seconds MY_USER_CACHE_LAST Set users for poll request between 31 to 60 seconds and each poll handler will see if i am between second 00 to 30 or 31 to 60 So if a poll come at 03:12:09(XX:YY:09) AM it will go to cache MY_USER_CACHE_FIRST and if request comes at 03:12:35(XX:YY:35) AM it will go to cache MY_USER_CACHE_LAST. And run your task at every 11:29:35((XX:YY:35) and 11:30:05((XX:YY:05), so 11:29:35(XX:YY:35) task will receive all users polled between 11:29:00(XX:YY:00) to 11:29:30(XX:YY:30). and 11:29:35(XX:YY:05) task will receive all users polled between 11:29:00(XX:(YY-1):00) to 11:29:30(XX:(YY-1):30). at the end of hour even XX will be XX-1 11:29:35 task will always read thye cache MY_USER_CACHE_FIRST and work on it as i explained before, in case u receive millions of records and divide into small task put seprately into small memcahe object etc and 11:30:05 task will always work on MY_USER_CACHE_LAST. I kept 5 second different between last entry going into cache and starting the task you can choose this difference as you want, may be just one second. Ravi. On Thu, Jun 24, 2
[google-appengine] What is a pattern for keeping track of current users in google app engine?
sorry typos! This still will not work at if you use two sets they both expire within 30 secs. So if i poll at 29 secs - in 1 sec my poll is lost - it should be kept for 29 seconds - if i poll on say 45 seconds my poll is lost in 15. The idea is to show a clear window of "who" polled in the last 30 seconds. Or that's how i understand the requirement So if a poll come at 03:12:09(XX:YY:09) AM it will go to cache MY_USER_CACHE_FIRST - #this person must exist for 30secs not 21# and if request comes at 03:12:35(XX:YY:35) AM it will go to cache MY_USER_CACHE_LAST. #this person must exist for 30 secs not 25# the only way to keep a track to the nearest second is with 30 poles 15 poles gives you a two second error 10 poles gives you a three second error 5 poles gives you a 6 second error 2 poles gives you a 15 second error 30 pole solution is the only way to do it if you want to use mcache in this way. Further on the note of reading the users. I don't know what the original developers was planing on doing with the data - he mentioned he wanted a task to do "something" every few seconds and their maybe a lot of users. On that note assuming he is NOT updating the users model - assuming maybe its an admin page with an Ajax update to show who is polling then each time a user poles(which i know would not be a task) he should store a dictionary with the data from the user he needs in the pole - when they poll - not load it every n second in his task. Its better to read it once - but again this is all guess work. Finally if he is updating the users table it would be just as easy to update a polled time and run a task that draws which ones polled in the time frame and do the updateamd as you rightly say perhaps do 100 at a time and make the task re-run when their are poles left. The solution i posed was merely the approach that would work in mcache and was a simple one. I hope this is now agreeable? From: Ravi Sharma To: google-appengine@googlegroups.com Sent: Thu, 24 June, 2010 19:40:50 Subject: Re: [google-appengine] Re:What is a pattern for keeping track of current users in google app engine? Yes i understand about that there might be lost in few entries. That you can fix in this solution by many ways..one of the way by keeping two sets then.( i thought of explaning it before but mail was getting longer :) ) Ok let me explain in detail But this to work the both cache need to have seprate Id like and expire time say 30 second or more. KeyValue description MY_USER_CACHE_FIRST Set users for poll request between 00 to 30 seconds MY_USER_CACHE_LAST Set users for poll request between 31 to 60 seconds and each poll handler will see if i am between second 00 to 30 or 31 to 60 So if a poll come at 03:12:09(XX:YY:09) AM it will go to cache MY_USER_CACHE_FIRST and if request comes at 03:12:35(XX:YY:35) AM it will go to cache MY_USER_CACHE_LAST. And run your task at every 11:29:35((XX:YY:35) and 11:30:05((XX:YY:05), so 11:29:35(XX:YY:35) task will receive all users polled between 11:29:00(XX:YY:00) to 11:29:30(XX:YY:30). and 11:29:35(XX:YY:05) task will receive all users polled between 11:29:00(XX:(YY-1):00) to 11:29:30(XX:(YY-1):30). at the end of hour even XX will be XX-1 11:29:35 task will always read thye cache MY_USER_CACHE_FIRST and work on it as i explained before, in case u receive millions of records and divide into small task put seprately into small memcahe object etc and 11:30:05 task will always work on MY_USER_CACHE_LAST. I kept 5 second different between last entry going into cache and starting the task you can choose this difference as you want, may be just one second. Ravi. On Thu, Jun 24, 2010 at 7:04 PM, Martin Webb wrote: @ping2ravi >this wont do what was required as if every-time you set mcache you will renew >the 30 second lifespan. further if mcache does clear on the 30 sec count a >poll 1 sec ago would be lost - that was the whole point of using 30 1 sec >poles as they all clear on deadline and don't get repeatedly refreshed? > > > >Regards > > >Martin Webb > > > >The information contained in this email is confidential and may contain >proprietary information. It is > meant solely for the intended recipient. Access to this email by anyone else > is unauthorised. If you are not the intended recipient, any disclosure, > copying, distribution or any action taken or omitted in reliance on this, is > prohibited and may be unlawful. No liability or responsibility is accepted if > information or data is, for whatever reason corrupted or does not reach its > intended recipient. No warranty is given that this email is free of viruses. > The views expressed in this email are, unless otherwise stated, those of the > author > > > > > > > ___
[google-appengine] What is a pattern for keeping track of current users in google app engine?
This still will not work at if you use two sets they both expire within 30 secs so if i poll at 29 secs - in 1 sec the my poll is removed it should be kept for 29 secods - if i poll on say 45 seconds my poll is lost in 15. The idea is to show a clear window of "who" polled in the last the seconds. Or that's how i understand the requirement So if a poll come at 03:12:09(XX:YY:09) AM it will go to cache MY_USER_CACHE_FIRST - #this person must exist for 30secs not 21# and if request comes at 03:12:35(XX:YY:35) AM it will go to cache MY_USER_CACHE_LAST. #this person must exist for 30 secs not 25# the only way to keep a track to the nearest second is with 30 poles 15 poles gives you a two second error 10 poles gives you a three second error 5 poles gives you a 6 second error 2 poles gives you a 15 second error 30 pole solution is the only way to do it if you want to use mcache in this way. Further on the note of reading the users. I don't know what the original developers was planing on doing with the data - he mentioned he wanted a task to do "something" every few seconds and their maybe a lot of users. On that note assuming he is NOT updating the users model - assuming maybe its an admin page with an Ajax update to show who is polling then each time a user poles(which i know would not be a task) he should store a dictionary with the data from the user he needs in the pole - when they poll - not load it every n second in his task. Its better to read it once - but again this is all guess work. Finally if he is updating the users table it would be just as easy to update a polled time and run a task that draws which ones polled in the time frame and do the updateamd as you rightly say perhaps do 100 at a time and make the task re-run when their are poles left. The solution i posed was merely the approach that would work in mcache and was a simple one. I hope this is now agreeable? From: Ravi Sharma To: google-appengine@googlegroups.com Sent: Thu, 24 June, 2010 19:40:50 Subject: Re: [google-appengine] Re:What is a pattern for keeping track of current users in google app engine? Yes i understand about that there might be lost in few entries. That you can fix in this solution by many ways..one of the way by keeping two sets then.( i thought of explaning it before but mail was getting longer :) ) Ok let me explain in detail But this to work the both cache need to have seprate Id like and expire time say 30 second or more. KeyValue description MY_USER_CACHE_FIRST Set users for poll request between 00 to 30 seconds MY_USER_CACHE_LAST Set users for poll request between 31 to 60 seconds and each poll handler will see if i am between second 00 to 30 or 31 to 60 So if a poll come at 03:12:09(XX:YY:09) AM it will go to cache MY_USER_CACHE_FIRST and if request comes at 03:12:35(XX:YY:35) AM it will go to cache MY_USER_CACHE_LAST. And run your task at every 11:29:35((XX:YY:35) and 11:30:05((XX:YY:05), so 11:29:35(XX:YY:35) task will receive all users polled between 11:29:00(XX:YY:00) to 11:29:30(XX:YY:30). and 11:29:35(XX:YY:05) task will receive all users polled between 11:29:00(XX:(YY-1):00) to 11:29:30(XX:(YY-1):30). at the end of hour even XX will be XX-1 11:29:35 task will always read thye cache MY_USER_CACHE_FIRST and work on it as i explained before, in case u receive millions of records and divide into small task put seprately into small memcahe object etc and 11:30:05 task will always work on MY_USER_CACHE_LAST. I kept 5 second different between last entry going into cache and starting the task you can choose this difference as you want, may be just one second. Ravi. On Thu, Jun 24, 2010 at 7:04 PM, Martin Webb wrote: @ping2ravi >this wont do what was required as if every-time you set mcache you will renew >the 30 second lifespan. further if mcache does clear on the 30 sec count a >poll 1 sec ago would be lost - that was the whole point of using 30 1 sec >poles as they all clear on deadline and don't get repeatedly refreshed? > > > >Regards > > >Martin Webb > > > >The information contained in this email is confidential and may contain >proprietary information. It is > meant solely for the intended recipient. Access to this email by anyone else > is unauthorised. If you are not the intended recipient, any disclosure, > copying, distribution or any action taken or omitted in reliance on this, is > prohibited and may be unlawful. No liability or responsibility is accepted if > information or data is, for whatever reason corrupted or does not reach its > intended recipient. No warranty is given that this email is free of viruses. > The views expressed in this email are, unless otherwise stated, those of the > author > > > > > > > __
[google-appengine] What is a pattern for keeping track of current users in google app engine?
I don't know if the solution allows a user to pole more than once in 30 seconds and i don't know if in fact you need to know if someone has polled multiple times in 30 seconds or keep an audit of all the historical poles - these questions would all change how this could and should be done. If you may have upto 1000 users polling in one 30 second segment then you would have a list of 33 user refs in a list stored in each pole. I also dont know what information must be stored when they pole - if you need time and user ref or just a ref or if in fact you need more information. mcache may loose data - though i am surprised some are saying this could happen in this scenario - so again i don't know how accurate this must be and if in fact a pole was lost from time time whether that would be a problem especially if of course the same users are polling continually and being added to other poles. speed over accuracy? Your first suggestion of using a model is still a good one tho 1000 entities being read every n seconds is hefty in my opinion. especially if you need to read user data from db and not just get the key. In the pole scenario a dictionary could be used and when the user poles their data read and stored in the pole in mcache at the time of polling as i have documented, which would save re-loading data from db when your task runs. That said in addition to my last post with the mcache 30 pole solution - a bug fix is below for working the pole no 1-30 i did mention i ran out of time on this method. The key for the pole would be made as follows second = #get the min interval of the current time from server if second>29 second=second-30 #this gives us 0-29 and then 0-29 again for one min then make the mcache key as key='poles%' %(second) If 30 poles are too many use 15 - one for each 2 second element or 10 one for each 3 second element - again their are accuracy concerns here 30 being the more accurate this would be down to what you need from this routine. If you have a need for the user data stored in the list - their is no need to merge it? just load each pole and iterate it doing whatever function you need to pursue? Their is no reason to merge the 30 poles for what reason? Thats just making extra work. This is the only obvious way i can see of using mcache to perform such a task - in comparison to using a model and fetching entities with a time zone. Martin -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] What is a pattern for keeping track of current users in google app engine?
Each of my users is polling the server every few seconds. I need to keep a list of the users that have polled in the last 30 seconds handy for a task I have queued to run every few seconds. The obvious way I see to do it is to update a datastore entry every time the user polls, and query the entries that have a timestamp within the last N seconds within my task queue. I can't imagine this scaling well. Any recommendations? Thanks. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
