For me the way I noticed it was by chance, also it seems it happened many times unnoticed. I was running a task for each namespace and each task would send an email. I noticed I received less emails than number of namespaces. I looked at logs there and couldn't find any errors nor even any evidence of these tasks ever executing, also i am sure they were sent to queue. As for max retries param, I believe I had a max set to 2-3.
Thanks On Friday, February 27, 2015 at 12:02:07 AM UTC, paynen wrote: > > In addition, re: anybody experiencing this issue, it would be very helpful > to include your queue config file, to make sure whether you're specifying > any kind of max retries parameter. > > On Thursday, February 26, 2015 at 8:54:02 AM UTC-5, husayt wrote: >> >> Hi @paynen, >> this is the problem. It's almost not possible to replicate externally, as >> it happens somewhere in internal appengine stack. >> >> and the main problem, as also explained by Kaan, it never hits logs. >> >> So there is not much we can do here as GAE users. This can be replicated >> only with access to internals of GAE. >> >> Can I also stress, that this is the number one issue on my list. I had a >> support case created and it didn't go forward because I couldn't replicate >> the problem. >> >> One thing I can say it more likely to happen when we have bursts of taks. >> >> >> Hope this helps, >> HG >> On Wednesday, February 25, 2015 at 10:52:06 PM UTC, paynen wrote: >>> >>> If anybody reading other than OP is also affected by this and can >>> provide minimally a reproducing example or an affected timeframe on a given >>> instance, this will be the minimum information needed to look into a >>> potential issue. >>> >>> I'm continuing to monitor this thread, and I hope we can get this >>> addressed as soon as possible, as soon as it's demonstrated/repro'd. >>> >>> On Monday, February 23, 2015 at 6:46:49 PM UTC-5, Kaan Soral wrote: >>>> >>>> rate: 500/s >>>>> >>>>> bucket_size: 100 >>>>> >>>>> retry_parameters: >>>>> >>>>> task_retry_limit: 6 >>>>> >>>>> min_backoff_seconds: 2 >>>>> >>>>> max_doublings: 3 >>>>> >>>>> >>>> Although my queue configuration is broad enough to handle occasional >>>> internal failures, I noticed and verified that the taskqueue leaves some >>>> tasks unexecuted >>>> ( 1% to 10%, happens when you burst tasks / run a mapreduce job >>>> [custom] - happens both with normal instances and basic_scaling/B4 >>>> instances ) >>>> >>>> I first noticed the issue when some operations that should have done >>>> were left undone >>>> >>>> Than I inspected the taskqueue execution with a custom routine that >>>> tracks / counts ingoing and executing tasks, a routine that I perfected >>>> long ago, and noticed the missing executions >>>> >>>> The issue isn't persistent, after a re-deployment and re-test, the same >>>> routine managed to traverse all the entities as it's supposed to >>>> >>>> TL;DR - some taskqueue tasks silently fail to execute, this should >>>> never happen, but it happens very frequently without any reason, causes >>>> damage and confusion >>>> >>> -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/275d8af0-b49f-4157-ab3a-164c82e57693%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
