Piyush, There are two options: TaskQueues <https://cloud.google.com/appengine/docs/standard/python/taskqueue/>on App Engine 1.9 and Cloud Pub/Sub <https://cloud.google.com/pubsub/docs/> everywhere else. Briefly, Pub/Sub offers fan-out (not just a queue) and is global (publish from any region, consume from any region). TaskQueues implement a queue (so no fan-out), are regional and are much better integrated with App Engine standard. It seems you may have more decisions to make here than just the messaging system:
1. How do you publish? "PUSH" suggests you may have some constraint on the message producer. Say it's a legacy piece of code that requires a webhook endpoint. Both services have REST APIs, but are better used with native client libraries. 2. Where your subscribers live. Since this is an AppEngine forum, I'll guess you are considering using GAE for processing the messages. Given your throughput, it might be more cost effective and faster to run consumers on GCE/GKE -- somewhere where you can run multi-threaded message consumers. At a steady 10K messages/second you are not going to benefit much from being able to scale down to 0. This is a guess so take it as such. There are other considerations k On Thursday, November 9, 2017 at 3:25:25 PM UTC-5, Piyush Katariya wrote: > > Hey Guys, > > Could you please help me decide the cloud tools which i can use to > achieve High Throughput Distributed Queue with Persistence ? > Distributed queue should support at least few thousands of Publishers > (PUSH) and subscribers (PULL). > Message ordering is not a priority for my application. > Message PULL latency observed by consumer after publisher is done with > PUSH should not be more than 10ms. > Message must not be lost at any cost. (at least once guarantee support or > Exactly once guarantee would be awesome) > Minumum required throughput should be 10K messages per second but should > be able scale if required. > > > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/8a687db8-05ab-4593-82d8-e408831f4bdc%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
