Hi Jeff, I am not sure I understand how putting the code in a transactional task helps me.
The simplified code is: Mail mailToSend = getMail(); SentStatus sentStatus = sendMailToExternalMailSystem(mailToSend); updateMailStatusInDatastore(mailToSend, sentStatus) The problem is that the datastore write "updateMailStatusInDatastore" might fail due to CME, and at this point the mail has already been sent to the external mail system. So even though I move this code to a transactional task, a failure in the datastore write will still lead me to a scenario where the mail has been sent, but the mail status has not been updated properly in my app. Thanks, -Louise Den torsdag den 30. marts 2017 kl. 20.30.45 UTC+2 skrev Jeff Schnitzer: > > There may be clock skew in the cluster; 15s is a lot but you can’t assume > that log entry timestamps are exact. > > You should send email by enqueueing a transactional task. Do not put > non-idempotent remote calls in your transactions (ideally, don’t put any > remote calls in transactions). > > For example, let’s say you want to email a receipt on purchase. Create a > deferred SendReceiptEmailTask that holds the purchase record key; it looks > up the record, formulates the email body, and makes the final call to the > email service. You can enlist this in your transaction that creates the > purchase record. This also has the benefit of moving all that work outside > of your transaction and shortening the critical section. > > There is a very small risk that tasks may execute twice, but it’s small > enough and harmless enough to ignore. I’ve sent millions of emails this way > with no complaints. And you don’t really have a choice anyway - I haven’t > found any email sending companies that allow you to specify an idempotency > key or some other way of guaranteeing once-and-only-once behavior. > > Cheers, > Jeff > > > > On Thu, Mar 30, 2017 at 3:44 AM, Louise Elmose Hedegaard < > louise...@gmail.com <javascript:>> wrote: > >> Hi Jeff, >> >> I have CME erratically - so I guess you are saying I should take a look >> at the look for the CME runs, and check whether the same EG's are accessed >> by other operations? >> It does seem that there might be a clash with other operations when the >> CME occurs. >> I am not sure about the timing though - there is a updateOrder operation >> which fails with CME at 14:59:46.684 and the send mail fails with CME >> at 15:01:06.583... >> When you say "Any change to that EG" it is the entire EG you mean, or >> only rows with the same id in the EG? >> >> I need to ensure that my transaction does not fail, as it will result in >> duplicate emails being sent if it does. >> There are other operations working on the same EG which comes randomly as >> they are caused by a webhook in another application. >> What is the best way to ensure that these webhook operations do not cause >> the send mail operation to get a CME? >> I can only think of very hacked and ugly solutions - e.g. to have a >> lock/switch which is on when the send mail operation is running, on only >> executing webhook operations when the lock/switch is off. >> >> Thanks, >> -Louise >> >> Den tirsdag den 28. marts 2017 kl. 22.11.15 UTC+2 skrev Jeff Schnitzer: >>> >>> When you load a key in a datastore transaction, that EG is enlisted in >>> the transaction. Any change to that EG by any other process in the system >>> will cause your commit to rollback with CME. Even if your transaction is >>> "read-only”. >>> >>> When I said “linear” I mean that if you have a quiet datastore (no other >>> activity) and you start a transaction and you perform a series of >>> operations and you commit the transaction, you should not see CME. >>> >>> If you see CME erratically, then you have contention in your system. >>> If you see CME consistently 100% of the time and you aren’t under heavy >>> load, then you have a bug in your transaction logic. >>> >>> Jeff >>> >>> On Tue, Mar 28, 2017 at 6:29 AM, Louise Elmose Hedegaard < >>> louise...@gmail.com> wrote: >>> >>>> Hi, >>>> >>>> I stumbled upon an instance in the log where the sendNow list is empty >>>> - but still the operation fails with ConcurrentModificationException. >>>> How is that possible - it is only the initial read 1) operation which >>>> is executed on the datastore. >>>> >>>> Thanks, >>>> -Louise >>>> >>>> >>>> Den tirsdag den 28. marts 2017 kl. 08.58.35 UTC+2 skrev Louise Elmose >>>> Hedegaard: >>>>> >>>>> Hi Adam, >>>>> >>>>> Why do you mention timeout - ConcurrentModificationException is not >>>>> related to timeouts is it? >>>>> >>>>> Thanks, >>>>> -Louise >>>>> >>>>> Den lørdag den 25. marts 2017 kl. 00.02.05 UTC+1 skrev Adam (Cloud >>>>> Platform Support): >>>>>> >>>>>> A great article that explains this is 'Timeouts due to write >>>>>> contention >>>>>> <https://cloud.google.com/appengine/articles/handling_datastore_errors#timeouts-due-to-write-contention> >>>>>> ': >>>>>> >>>>>> *Writes to a single entity group are serialized by the App Engine >>>>>>> datastore, and thus there's a limit on how quickly you can update one >>>>>>> entity group. In general, this works out to somewhere between 1 and 5 >>>>>>> updates per second.* >>>>>> >>>>>> >>>>>> Writing to the same entity twice successively is enough to cause a >>>>>> ConcurrentModificationException. Strategies to avoid this are discussed >>>>>> in 'Avoiding >>>>>> datastore contention >>>>>> <https://cloud.google.com/appengine/articles/scaling/contention>'. >>>>>> >>>>>> On Friday, March 24, 2017 at 3:18:28 AM UTC-4, Louise Elmose >>>>>> Hedegaard wrote: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> In my app I have the following logic (pseudoCode): >>>>>>> >>>>>>> List<NonSentMails> nonSentMails= getNonSentMails();//1) datastore >>>>>>> read >>>>>>> >>>>>>> List<NonSentMails> sendNow = new ArrayList<NonSentMails>(); >>>>>>> for each nonSentMail in nonSentMails >>>>>>> if (sendNow(nonSentMail)){ >>>>>>> sendNow.add(nonSentMail); >>>>>>> } >>>>>>> >>>>>>> for each nonSentMail in sendNow >>>>>>> SentStatus sentStatus = sendMailViaRestApi(); >>>>>>> create maillog in datastore with sent status//2) datastore write >>>>>>> - new entry >>>>>>> nonSentMail.setSentStatus(sentStatus); >>>>>>> >>>>>>> update(sendNow) //3) datastore write >>>>>>> >>>>>>> >>>>>>> When comitting the above I get: >>>>>>> java.util.ConcurrentModificationException: too much contention on >>>>>>> these datastore entities. please try again. >>>>>>> >>>>>>> This is a problem, as the operation is not idempotent - when I send >>>>>>> a mail via a REST API the mail is sent immediately, and I want the >>>>>>> status >>>>>>> of my maillog and NonSentMail objects to reflect this. >>>>>>> >>>>>>> I do not understand why this error occurs. >>>>>>> There are only three operations on the datastore: >>>>>>> 1) The initial read of non sent mails >>>>>>> 2) The creation of new mail log entries >>>>>>> 3) The update of the status of non sent mails >>>>>>> >>>>>>> 2) should never lead to ConcurrentModificationException as far as I >>>>>>> understand it. 1) and 3) operates on the same entities, but 1) only >>>>>>> reads, >>>>>>> so I cannot see any problem here either. >>>>>>> Furthermore there is currently only one user of my app, so there >>>>>>> should be no other threads accessing the data. >>>>>>> Note that the elements in sendNow might have the same parents, but >>>>>>> again I do not believe this should be a problem? >>>>>>> >>>>>>> Can you please help me understand why the above code can lead to a >>>>>>> ConcurrentModificationException? >>>>>>> >>>>>>> Thanks, >>>>>>> -Lull >>>>>>> >>>>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Google App Engine" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to google-appengi...@googlegroups.com. >>>> To post to this group, send email to google-a...@googlegroups.com. >>>> Visit this group at https://groups.google.com/group/google-appengine. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/google-appengine/02ac6e5d-e70d-4da8-9f51-2ac68aefc3d7%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/google-appengine/02ac6e5d-e70d-4da8-9f51-2ac68aefc3d7%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to google-appengi...@googlegroups.com <javascript:>. >> To post to this group, send email to google-a...@googlegroups.com >> <javascript:>. >> Visit this group at https://groups.google.com/group/google-appengine. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/google-appengine/ea889f39-59c1-4493-ab7b-f4db8d25b2a8%40googlegroups.com >> >> <https://groups.google.com/d/msgid/google-appengine/ea889f39-59c1-4493-ab7b-f4db8d25b2a8%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/b546a386-0365-4749-9f64-faee69adbe72%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
