Re: [google-appengine] TransactionFailedError: Is the transaction committed or not?
It won't help avoid the collision. It will let you keep track of previous transactions. On Jul 29, 2011 8:20 PM, "vlad" wrote: > > Joushua, > > I was thinking of putting child entities approach as well. The question is does writing a new child entity blocks the whole entity group? If answer is yes. Then you get no benefit from that other than keeping meticulous track of what is happening. I would consider that approach if it lets me avoid transaction collisions as well. > > -- > You received this message because you are subscribed to the Google Groups "Google App Engine" group. > To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/TQa0MN-_vtEJ. > > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] TransactionFailedError: Is the transaction committed or not?
Joushua, I was thinking of putting child entities approach as well. The question is does writing a new child entity blocks the whole entity group? If answer is yes. Then you get no benefit from that other than keeping meticulous track of what is happening. I would consider that approach if it lets me avoid transaction collisions as well. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/TQa0MN-_vtEJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] TransactionFailedError: Is the transaction committed or not?
Hi Joshua, As robert said, making a process idempotent depends mostly on the app logic and data model. IMO for the example case you need more of a context of how the counter is been use to make the use of the counter in a transaction idempotent. (remember: the example just illustrates how to make use of a transaction, it does not handles any retry if the transaction failed or not) As you said: making a separate child model, which is parent by the object you are transacting on, and inside the transaction query to see if its already in there, Its a good solution to make your transaction idempotent, and I don't see a problem why this approach wont scale. If you do an ancestor query on that kind it would be very fast. With this approach, the code inside the transaction should do something like: 1) read child entity 2) if entity not processed, do the transaction logic. else: no nothing. FYI GAE support transactions, because either the whole transaction succeeds or fails. Whether you know about the outcome or not is different. Although making process idempotent is somewhat a complicated logic, here the better solution would be to figure out why the exception is occurring. Mostly because this type of exception are not common. You don't need to "only" write idempotent transactions, you should make them idempotent, unless it does not matter if they succeed or not, as I said before this type of exception are not common so there are many cases where 100% data retention is not as important. On Thursday, July 28, 2011 7:02:53 AM UTC-7, Joshua Smith wrote: > > The problem is that google transactions can report an exception, and then > go ahead and succeed anyway. > > So the docs recommend that you only write idempotent transactions, which is > a completely silly suggestion. I've yet to see a single example of how one > might write an idempotent transaction. (Unless, I suppose, you create a > separate child model in the database which is parented by the object you are > transacting on, and then you query the list of children every time you retry > your transaction to see if its already in there, but that won't scale.) > > I contend that a DB that cannot tell you reliably whether a transaction > succeeded for failed does not support transactions. > > GAE can essentially report 3 possible results from a transaction: > - Definitely succeeded > - Definitely failed > - Beats me > > I contend that third possible result makes it impossible to write software > that relies on transactions. > > Therefore, GAE doesn't support transactions. > > On Jul 27, 2011, at 8:58 PM, Tim Hoffman wrote: > > If you always get modify put within a transaction how would it be > unreliable? > > Rgds > > Tim > > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/mP_8kv_-LlMJ. > To post to this group, send email to google-a...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengi...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/FgjOX6rQpu4J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] TransactionFailedError: Is the transaction committed or not?
I should have mentioned that the re-fetch should be done in a transaction, which should take away the "eventually" part since from what I've gathered I think you're reading directly from the master transaction log. As for the continuous timing out due to datastore issues then you'd have to fallback to the task queue as Joshua has stated. It's not a perfect solution but oh well. On Thu, Jul 28, 2011 at 10:48 AM, Joshua Smith wrote: > > > 6.) if exception is generated, then re-query the entity if the entity has > "our" unique timestamp in its list then the transaction really did succeed, > if not then the transaction really did fail. > > Could work, except that you'll need to do #6 with a task queue task, > because in the case of an exception, they only say it might "eventually" > make it into the database. So if you just requery and it isn't there, then > you don't know that it didn't succeed… you just know it didn't succeed > *yet*. And since the exception which leads to this problem is > "InternalError", and we know when those start coming up you can expect to > see them continue for a long time, you're going to have a lot of stuff in > that queue. > > So that means you need to throw this check into the task queue, and keep > checking it for a while. Which probably means you need to have a really big > list of "recent" transactions (not just 10). > > This hack is similar to what I suggested, which is to create a transaction > record in the database as a child of the entity you are updating in the > transaction. When you get the exception, you wait a while (how long???) and > then see if the transaction record exists using an ancestor query. If it > does, then the transaction succeeded, and if not, then it didn't. This > avoids having to add goofy hacky attributes to the thing you are updating, > and in principle, you can schedule a task to delete any transaction records > for successful transactions (or periodically sweep up old ones with a cron > job). > > So, I suppose, it is possible to create a framework that can really do > transactions using the building blocks GAE has provided us, but to say that > GAE supports transactions on its own is a stretch. > > -Joshua > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] TransactionFailedError: Is the transaction committed or not?
> 6.) if exception is generated, then re-query the entity if the entity has > "our" unique timestamp in its list then the transaction really did succeed, > if not then the transaction really did fail. Could work, except that you'll need to do #6 with a task queue task, because in the case of an exception, they only say it might "eventually" make it into the database. So if you just requery and it isn't there, then you don't know that it didn't succeed… you just know it didn't succeed *yet*. And since the exception which leads to this problem is "InternalError", and we know when those start coming up you can expect to see them continue for a long time, you're going to have a lot of stuff in that queue. So that means you need to throw this check into the task queue, and keep checking it for a while. Which probably means you need to have a really big list of "recent" transactions (not just 10). This hack is similar to what I suggested, which is to create a transaction record in the database as a child of the entity you are updating in the transaction. When you get the exception, you wait a while (how long???) and then see if the transaction record exists using an ancestor query. If it does, then the transaction succeeded, and if not, then it didn't. This avoids having to add goofy hacky attributes to the thing you are updating, and in principle, you can schedule a task to delete any transaction records for successful transactions (or periodically sweep up old ones with a cron job). So, I suppose, it is possible to create a framework that can really do transactions using the building blocks GAE has provided us, but to say that GAE supports transactions on its own is a stretch. -Joshua -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] TransactionFailedError: Is the transaction committed or not?
The problem is that google transactions can report an exception, and then go ahead and succeed anyway. So the docs recommend that you only write idempotent transactions, which is a completely silly suggestion. I've yet to see a single example of how one might write an idempotent transaction. (Unless, I suppose, you create a separate child model in the database which is parented by the object you are transacting on, and then you query the list of children every time you retry your transaction to see if its already in there, but that won't scale.) I contend that a DB that cannot tell you reliably whether a transaction succeeded for failed does not support transactions. GAE can essentially report 3 possible results from a transaction: - Definitely succeeded - Definitely failed - Beats me I contend that third possible result makes it impossible to write software that relies on transactions. Therefore, GAE doesn't support transactions. On Jul 27, 2011, at 8:58 PM, Tim Hoffman wrote: > If you always get modify put within a transaction how would it be unreliable? > > Rgds > > Tim > > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/mP_8kv_-LlMJ. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] TransactionFailedError: Is the transaction committed or not?
If you always get modify put within a transaction how would it be unreliable? Rgds Tim -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/mP_8kv_-LlMJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] TransactionFailedError: Is the transaction committed or not?
Can you give an example? Because that approach would seem to be unreliable in a two-writer scenario. On Jul 26, 2011, at 10:11 PM, Robert Kluin wrote: > Well, I generally put revision numbers on my entities. In my > transctions, before the update, I check that the revision number > currently on the entity matches what I expect. If it doesn't then I > know the update is out-of-date. > > There are other solutions too, depending on the app and data model. > > > Robert > > > > On Tue, Jul 26, 2011 at 15:03, Joshua Smith wrote: >> On this topic, nobody ever answered this question: >> >> http://code.google.com/appengine/docs/python/datastore/transactions.html >> >> First it says, "Make sure your transactions are idempotent" and then it >> gives an example which isn't. >> >> I'm not sure it's possible to do the task in that example correctly if you >> cannot tell whether a transaction succeeded or failed when it throws an >> exception. I just tried sketching out a solution that stored a transaction >> ID in the model, but that won't work because there could be multiple >> writers. The whole thing seems rather intractable, and the idea that you >> cannot tell whether a transaction succeeded or failed violates the principle >> of least surprise for anyone who's ever used a database! >> >> Can some googlers weigh in, and explain how, for example, the example in the >> documentation could be implemented correctly? >> >> >> So, can you? >> I think that the idea of making transactions idempotent is nonsense. I >> don't think it is going to be possible in many cases. >> Either you support transactions, or you don't. By my reading, GAE doesn't. >> -Joshua >> On Jul 26, 2011, at 3:40 PM, Jose Montes de Oca wrote: >> >> Hi Pol, >> What this meas is that even if a transaction throws an exception this does >> not means the transaction failed, thats why you need to make your datastore >> transaction idempotent. So if you retry a transaction because it throws an >> exception, your transaction needs to "check" if the last >> transaction committed successfully or not. >> Best, >> Jose Montes de Oca >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To view this discussion on the web visit >> https://groups.google.com/d/msg/google-appengine/-/bm54SrfYFdAJ. >> To post to this group, send email to google-appengine@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to google-appengine@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] TransactionFailedError: Is the transaction committed or not?
Well, I generally put revision numbers on my entities. In my transctions, before the update, I check that the revision number currently on the entity matches what I expect. If it doesn't then I know the update is out-of-date. There are other solutions too, depending on the app and data model. Robert On Tue, Jul 26, 2011 at 15:03, Joshua Smith wrote: > On this topic, nobody ever answered this question: > > http://code.google.com/appengine/docs/python/datastore/transactions.html > > First it says, "Make sure your transactions are idempotent" and then it > gives an example which isn't. > > I'm not sure it's possible to do the task in that example correctly if you > cannot tell whether a transaction succeeded or failed when it throws an > exception. I just tried sketching out a solution that stored a transaction > ID in the model, but that won't work because there could be multiple > writers. The whole thing seems rather intractable, and the idea that you > cannot tell whether a transaction succeeded or failed violates the principle > of least surprise for anyone who's ever used a database! > > Can some googlers weigh in, and explain how, for example, the example in the > documentation could be implemented correctly? > > > So, can you? > I think that the idea of making transactions idempotent is nonsense. I > don't think it is going to be possible in many cases. > Either you support transactions, or you don't. By my reading, GAE doesn't. > -Joshua > On Jul 26, 2011, at 3:40 PM, Jose Montes de Oca wrote: > > Hi Pol, > What this meas is that even if a transaction throws an exception this does > not means the transaction failed, thats why you need to make your datastore > transaction idempotent. So if you retry a transaction because it throws an > exception, your transaction needs to "check" if the last > transaction committed successfully or not. > Best, > Jose Montes de Oca > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/bm54SrfYFdAJ. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] TransactionFailedError: Is the transaction committed or not?
On this topic, nobody ever answered this question: > http://code.google.com/appengine/docs/python/datastore/transactions.html > > First it says, "Make sure your transactions are idempotent" and then it gives > an example which isn't. > > I'm not sure it's possible to do the task in that example correctly if you > cannot tell whether a transaction succeeded or failed when it throws an > exception. I just tried sketching out a solution that stored a transaction ID > in the model, but that won't work because there could be multiple writers. > The whole thing seems rather intractable, and the idea that you cannot tell > whether a transaction succeeded or failed violates the principle of least > surprise for anyone who's ever used a database! > > Can some googlers weigh in, and explain how, for example, the example in the > documentation could be implemented correctly? > So, can you? I think that the idea of making transactions idempotent is nonsense. I don't think it is going to be possible in many cases. Either you support transactions, or you don't. By my reading, GAE doesn't. -Joshua On Jul 26, 2011, at 3:40 PM, Jose Montes de Oca wrote: > Hi Pol, > > What this meas is that even if a transaction throws an exception this does > not means the transaction failed, thats why you need to make your datastore > transaction idempotent. So if you retry a transaction because it throws an > exception, your transaction needs to "check" if the last transaction > committed successfully or not. > > Best, > Jose Montes de Oca > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/bm54SrfYFdAJ. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] TransactionFailedError: Is the transaction committed or not?
Hi Pol, What this meas is that even if a transaction throws an exception this does not means the transaction failed, thats why you need to make your datastore transaction idempotent. So if you retry a transaction because it throws an exception, your transaction needs to "check" if the last transaction committed successfully or not. Best, Jose Montes de Oca -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/bm54SrfYFdAJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] TransactionFailedError: Is the transaction committed or not?
Hi Pol, Generally you will probably want to execute it again. Robert On Mon, Jul 25, 2011 at 03:53, Pol wrote: > Hi, > > The exception says: TransactionFailedError: The transaction could not > be committed. Please try again. > > The doc at > http://code.google.com/appengine/docs/python/datastore/transactions.html > says: You can receive Timeout, TransactionFailedError, or > InternalError exceptions in cases where transactions have been > committed and eventually will be applied successfully. > > So if you get TransactionFailedError, to you need to execute the > transaction again or will it automatically be applied later? > > - Pol > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] TransactionFailedError: Is the transaction committed or not?
Hi, The exception says: TransactionFailedError: The transaction could not be committed. Please try again. The doc at http://code.google.com/appengine/docs/python/datastore/transactions.html says: You can receive Timeout, TransactionFailedError, or InternalError exceptions in cases where transactions have been committed and eventually will be applied successfully. So if you get TransactionFailedError, to you need to execute the transaction again or will it automatically be applied later? - Pol -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
