Why not leave the rate at 1/s, but set max_concurrent_requests to 1 as Nick suggested?
On Tue, Feb 21, 2012 at 12:56, James Gilliam <jimgill...@gmail.com> wrote: > Yes ... I saw the 5 per sec in the admin ... > > Also, I just realized why I think gae created so many instances when i > had the queue at 1/s ... I had assumed the tasks would take much less > than a second (which was true at the beginning); but as the number of > entity writes continued, the latency for the writes got larger and > eventually took longer than a second ... at which point gae started > new instances to keep up with the 1 sec rate ... i had assumed it > would not consider starting a new task until the previous one in the > queue finished -- this appears to not be the semantics > > My solution is to slow down the tasks even further ... to say 15 / > min ... so that new instances will not be created ... > > On Feb 19, 12:21 pm, Nicholas Verne <nve...@google.com> wrote: >> James, >> >> The misreporting of number of tasks in a queue is a known bug that >> we're working on. It causes undue alarm for our users. >> >> How do you know that GAE changed the target rate to 5 / sec ? Did you >> observe this in the admin console? >> >> The queue's rate is the rate at which tasks are dispatched to your >> app. If your queue is set to run 1 / sec and your task latency is >> longer than 1 second, you should expect to have more instances spin >> up. >> >> If you wish to prevent more than one task running at a time, set your >> queue's max_concurrent_requests to 1. >> >> Nick Verne >> >> >> >> >> >> >> >> On Mon, Feb 20, 2012 at 3:35 AM, James Gilliam <jimgill...@gmail.com> wrote: >> > I am using task queue in python. >> >> > I added 10,000 tasks for a queue set to run at 1/sec and a bucket of >> > 1 ... they were short tasks and I didn't want them to run >> > immediately ... I didn't even want them to run multiples per second >> > even though the latency was low ... it was ok if it took hours as I >> > knew it would at 1/sec. I figured it would create a second instance >> > as the site was continuing to run ... but I was pretty surprised with >> > what actually happen. >> >> > At first the queue started processing fine ... approx 60 tasks running >> > per minute ... >> >> > Then GAE decided to run them at a faster rate, change the target rate >> > to 5 / sec and created lots of new instances ... >> >> > When there were about 3000 tasks left ... GAE went crazy and said >> > there were nearly 200,000 tasks in the queue ... >> >> > As far as I could tell, the number of tasks being reported was just an >> > anomaly ... but it was alarming given that GAE was creating instances >> > like crazy. >> >> > The tasks would add new records (sometimes) and I noticed that the >> > latency of these tasks increased the longer the tasks ran. At first >> > just 200ms ... by the end nearly 1400 ms. >> >> > So -- >> >> > 1. The most important question -- Why did GAE not respect my desire to >> > run the queue at 1 / sec ? Is there a way I can stop GAE from >> > overriding my wishes? >> >> > 2. Was the latency increasing because the records being added were >> > slowing new records from being added? >> >> > 3. The invalid number of tasks seems like a harmless bug since new >> > phantom tasks were not actually created (as far as I know). >> >> > -- >> > You received this message because you are subscribed to the Google Groups >> > "Google App Engine" group. >> > To post to this group, send email to google-appengine@googlegroups.com. >> > To unsubscribe from this group, send email to >> > google-appengine+unsubscr...@googlegroups.com. >> > For more options, visit this group >> > athttp://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
