Thanks for the clarification, It looks like I will have to keep hitting datastore for my use case.
BTW: Can Fantasm benefit from making lock transactions using cloud spanner <https://cloud.google.com/spanner/> instead of datastore? or is datastore faster for small transactions like the ones performed by lock.py? Thanks On Monday, 9 October 2017 10:28:04 UTC-3, Jason Collins wrote: > > No. You can't make that assumption about task queue delivery, though it's > very, very good. This is why Fantasm had to go the extra distance with > Datastore transactions. > > BTW, Fantasm allows you to turn off the Datastore run-once check and rely > solely on task queue because in practice, task queue does a great job. Just > not a perfect one. So you have to choose between the small performance hit > of Datastore transaction, or very small chance of duplicate task delivery. > > BTW I can't recall the named task deduplication ever failing when > enqueuing a task. > On Mon, Oct 9, 2017 at 4:48 AM Nilson Pontello <nilson....@gmail.com > <javascript:>> wrote: > >> Thanks Jason/George, >> >> My code does a job very similar to lock.py >> <https://github.com/iki/fantasm/blob/master/src/fantasm/lock.py#L150> >> (this is what I am trying to avoid). >> >> So it looks like the secret is around task queues. Can I assume that >> enqueued tasks will be delivered *just once *(if my endpoint returns 200 >> OK)? >> >> If yes then my problem is solved. But if it behaves similar to pubsub >> which guarantees "At-Least-Once Delivery" then it will be useless for my >> case. >> >> BTW: If those named tasks are hitting datastore for deduplication then >> lock.py's approach is still valid and its performance will be as good as >> named tasks. >> >> Thanks >> >> >> On Saturday, 7 October 2017 23:45:25 UTC-3, Jason Collins wrote: >>> >>> Fantasm! That's a blast from the past! >>> >>> I'm one of the original authors of that package. It does indeed help to >>> make things idempotent, but at the end of the day, it relies on a datastore >>> transaction (fronted by memcache for some performance gains): >>> https://github.com/iki/fantasm/blob/master/src/fantasm/lock.py#L150 >>> >>> Datastore transactions are one of the few things you can "count on" in a >>> distributed system like App Engine. >>> >>> Another tool that is often helpful are named tasks, which can help >>> ensure that you don't requeue an already queued task (e.g., in the event of >>> a retry): >>> https://cloud.google.com/appengine/docs/standard/python/taskqueue/push/creating-tasks#naming_a_task >>> >>> Finally, keep in mind that you can also enqueue a task transactionally >>> with a Datastore transaction: >>> https://cloud.google.com/appengine/docs/standard/python/ndb/transactions#python_Transactional_task_enqueuing >>> >>> With some creativity, you can get a long way with these basic building >>> blocks. >>> >>> >>> On Friday, 6 October 2017 16:39:14 UTC-7, George (Cloud Platform >>> Support) wrote: >>>> >>>> Hello Nilson, >>>> >>>> Among other sources, you may benefit from reading the "Implementing >>>> Workflows on Google App Engine with Fantasm" article >>>> <https://cloud.google.com/appengine/articles/fantasm>. Fantasm is an >>>> excellent launch pad for building an understanding of how to decompose a >>>> workflow into the appropriate chunk size as well as gaining a solid >>>> footing >>>> in how to build idempotent states. >>>> >>>> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "Google App Engine" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/google-appengine/p8Xcap4eaKU/unsubscribe >> . >> To unsubscribe from this group and all its topics, send an email to >> google-appengi...@googlegroups.com <javascript:>. >> To post to this group, send email to google-a...@googlegroups.com >> <javascript:>. >> Visit this group at https://groups.google.com/group/google-appengine. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/google-appengine/4ccbccfa-e90a-4ac0-a125-503ba343e2ab%40googlegroups.com >> >> <https://groups.google.com/d/msgid/google-appengine/4ccbccfa-e90a-4ac0-a125-503ba343e2ab%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/03b67df6-f61d-41db-9847-1f839d201ee7%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
