Every minute or so my app creates some data and needs to send it out to more than 1000 remote servers via URL Fetch callbacks. This is done asynchronously using NDB. The time lag between creating the data and sending it to the remote servers via the callback should be roughly less than 5 seconds.
My initial thought is to use the Pipeline API to fan out URL Fetch requests to different task queues. Unfortunately task queues are not guaranteed to be executed in a timely fashion. Therefore from requesting a task queue start to it actually executing could take minutes to hours. From previous experience this gap is regularly over a minute so is not necessarily appropriate. Is there any way from within App Engine to achieve what I want? Maybe you know of an outside service that can do the fan out in a timely fashion? The question is on Stack Overflow: http://stackoverflow.com/questions/15829813/how-to-fan-out-url-fetch-requests-in-a-timely-fashion -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
