[google-appengine] Re: How stable is pull queue task tagging for arbitrary tags?
I'm currently using the pull queue with tags, and have experienced TranscientError in the past. Thanks to support cases still being there I can go back in time with precise figures :) Here are my usage figures if it can help: Currently: Daily average 7-8 tasks/second enqueued over 2 tags, with a spike at 37 tasks/second over 5 minutes. One worker per tag that wakes up every 30 minutes to lease them. No transcient error. Oct 2012 (they probably improved the service since): We were using many different tags (basically regrouping events per user with several million users). 1,800 tasks/sec enqueued, can peak at 2,500. Up to 100 requests/seconds that will lease tasks from the same pull task queue - but on different tags. = lots of TranscientError and DeadlineExceededError. To mitigate this the solution was to spread the tasks over multiple task queues: 20 pull task queues instead of one, each pull task queue had 35 tasks added by second, and 4 requests per second were leasing and deleting tasks from that same queue. I did still have regularly DeadlineExceeeded errors mainly about the call to delete tasks. Retrying at once solves it most of the time. I still found a few TranscientError (mostly on tasks delete too) but in such rate that made them acceptable. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
[google-appengine] Re: How stable is pull queue task tagging for arbitrary tags?
I forgot to mention - but you may realize if you do the math - the figures about when I split to 20 task queues were obtained while testing with traffic split, so it did not get the whole 1,800req/sec traffic but a subset of it. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
[google-appengine] Re: How stable is pull queue task tagging for arbitrary tags?
Thanks so much, the information is extremely useful I was unsure about the need for multiple pull queues, but it seems to be a must. I do use the method for regular queues, where the limits are clear On Wednesday, March 12, 2014 10:27:23 PM UTC+2, Alexis wrote: I forgot to mention - but you may realize if you do the math - the figures about when I split to 20 task queues were obtained while testing with traffic split, so it did not get the whole 1,800req/sec traffic but a subset of it. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
[google-appengine] Re: How stable is pull queue task tagging for arbitrary tags?
I've checked the performance of my datastore based task pipeline, the performance seems to be very poor, there are many missed/old tasks (db entities) - probably caused by query delays - as I query for tasks, and the delay should be ~30 seconds at most for the system to work, doesn't seem to be the case, always Anyway, let's say the pull queue tags are random strings - and I add and pull 100's of items every seconds dynamically, would it work? I've experienced taskqueue.TransientError situations with non-tagged but extremely heavy usage, happened when I was pulling and deleting only 1 task, by multiple workers, however I have a feeling the above scenario, with infinite tags, might work better, as the incoming rate would be high, but the tasks would be pulled in batches and not so frequently Anyway the above explanation might not be sufficient to explain my usecase and needs, however I would appreciate any usage experience around pull queues, thanks in advance -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
