Hi there, Our (python) app uses a pull queue to send work to dynamic backend instances. The backends sit in a tight loop polling for new work, 5 tasks at a time. As long as tasks are coming through, the backends periodically send 'keepalive' tasks to themselves via a push queue. Once the pull queue is exhausted, the backends spin down since they go idle and no keepalive tasks come through.
Up until yesterday, this system was working fine, with the caveat that lease_tasks was throwing large numbers of TransientErrors, which we would catch and ignore. As of today, lease_tasks has started to fail silently - there are over 30k tasks in the pull queue with ETAs in the past, yet lease_tasks intermittently (but frequently) returns 0 results. It's happening often enough that there are insufficient keepalive tasks, meaning the backends die off even though there is tons of work to do. Anyone else seeing this? Did something change in the taskqueue implementation? Thanks, -Abhijit -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/mxlK_vp0TuEJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
