[google-appengine] Re: TransactionFailedError: Is the transaction committed or not?

2011-07-30 Thread Heiko Roth


On 26 Jul., 07:23, Robert Kluin robert.kl...@gmail.com wrote:
 Hi Pol,
   Generally you will probably want to execute it again.


No. I've got the problem for time registration. We got an exception
that the transaction failed, so we do it again which can result in
duplicate (and more) datastore entries which is bad. Because now there
are duplicate bookings and so duplicate working time...

It's a disaster that the program gets a transaction failed while the
transaction didn't doing another insert resulting in a mess.
But ignoring the exception can result in now datastore entry.

So we have catch the exception, do a select and repeat the insert if
it didn't happen by now.
And that we have to do for any insert ..

I'm frustrated.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



Re: [google-appengine] Re: TransactionFailedError: Is the transaction committed or not?

2011-07-30 Thread Robert Kluin
If you have a way to precompute something to use as a key-name this is
significantly less of an issue.  The transaction repeating would not
result in duplicate record being created that way.


Robert






On Sat, Jul 30, 2011 at 12:39, Heiko Roth r...@egotec.com wrote:


 On 26 Jul., 07:23, Robert Kluin robert.kl...@gmail.com wrote:
 Hi Pol,
   Generally you will probably want to execute it again.


 No. I've got the problem for time registration. We got an exception
 that the transaction failed, so we do it again which can result in
 duplicate (and more) datastore entries which is bad. Because now there
 are duplicate bookings and so duplicate working time...

 It's a disaster that the program gets a transaction failed while the
 transaction didn't doing another insert resulting in a mess.
 But ignoring the exception can result in now datastore entry.

 So we have catch the exception, do a select and repeat the insert if
 it didn't happen by now.
 And that we have to do for any insert ..

 I'm frustrated.

 --
 You received this message because you are subscribed to the Google Groups 
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to 
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



[google-appengine] Re: TransactionFailedError: Is the transaction committed or not?

2011-07-29 Thread Pol
Hi Jose,

 2) There is no fake-real-transaction, for example, one of the other uncommon
 exception that someone could get in the commit() would be Timeout exception.
 With any distributed system, unless you are willing to wait for an answer
 forever, there is always the possibility that you will get a timeout before
 the remote system responds with the result of the operation you requested
 (this is true in all databases), what GAE warranty is that the transaction
 either commits successfully or it does not, so if you implement some retry
 logic in transactions, having it idempotent will warranty you don't do the
 transaction twice.

Just to be sure, are you saying that for a timeout exception the
transaction has actually been committed or not? In other words, should
a Timeout exception be dealt with the same way a
TransactionFailedError is?

I observe Timeout exceptions now and then where there is too much
contention on the same entity group. But I've only seen a
TransactionFailedError once in 2 months.

 But as I said before, this exceptions are uncommon and it would be good to
 know what are you doing inside the transaction that is causing those
 transactions. Timeout exception could be related to master slaves issues (in
 a M/S app) this could go away by migrating to High replication datastore.

I'm using the HR datastore already, but I'm not surprised with the
Timeouts as in my case there can sometimes, but rarely, be close to 10
transactions / seconds on the same entity group.

 Hope this helps clarifies some points.

Thanks much for your detailed answers indeed!

- Pol

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



[google-appengine] Re: TransactionFailedError: Is the transaction committed or not?

2011-07-29 Thread Tim Hoffman


On Friday, July 29, 2011 10:17:30 PM UTC+8, Pol wrote:

 Hi Jose, 

  2) There is no fake-real-transaction, for example, one of the other 
 uncommon 
  exception that someone could get in the commit() would be Timeout 
 exception. 
  With any distributed system, unless you are willing to wait for an answer 

  forever, there is always the possibility that you will get a timeout 
 before 
  the remote system responds with the result of the operation you requested 

  (this is true in all databases), what GAE warranty is that the 
 transaction 
  either commits successfully or it does not, so if you implement some 
 retry 
  logic in transactions, having it idempotent will warranty you don't do 
 the 
  transaction twice. 

 Just to be sure, are you saying that for a timeout exception the 
 transaction has actually been committed or not? In other words, should 
 a Timeout exception be dealt with the same way a 
 TransactionFailedError is? 


No.  I believe he was saying if you get a timeout it just means you didn't 
get a response 
back in time.  So the transaction could have completed or failed. You don't 
know.
What you can be certain of is either everything in the transaction succeeded 
or failed.

 


 I observe Timeout exceptions now and then where there is too much 
 contention on the same entity group. But I've only seen a 
 TransactionFailedError once in 2 months. 

 I get TransactionFailedError quite a bit.

The transaction could not be committed. Please try again.
Traceback (most recent call last):
  File 
/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/__init__.py,
 line 702, in __call__
handler.post(*groups)
  File 
/base/python_runtime/python_lib/versions/1/google/appengine/ext/deferred/deferred.py,
 line 269, in post
run(self.request.body)
  File 
/base/python_runtime/python_lib/versions/1/google/appengine/ext/deferred/deferred.py,
 line 131, in run
return func(*args, **kwds)
  File 
/base/python_runtime/python_lib/versions/1/google/appengine/ext/deferred/deferred.py,
 line 174, in invoke_member
return getattr(obj, membername)(*args, **kwargs)
  File 
/base/data/home/apps/q-tracker/0-9-9-9.352045897303914982/qtrack/models/contract.py,
 line 268, in defer_create_worksheet
db.run_in_transaction(contract.create_worksheet,start_of_week(week))
  File 
/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py, 
line 2187, in RunInTransaction
DEFAULT_TRANSACTION_RETRIES, function, *args, **kwargs)
  File 
/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py, 
line 2294, in RunInTransactionCustomRetries
'The transaction could not be committed. Please try again.')
TransactionFailedError: The transaction could not be committed. Please try 
again.


But this is running in a deferred task, and the task is automatically put 
back into the queue, so I know it will eventually get there.
I am running on M/S.

Rgds

Tim

 

  But as I said before, this exceptions are uncommon and it would be good 
 to 
  know what are you doing inside the transaction that is causing those 
  transactions. Timeout exception could be related to master slaves issues 
 (in 
  a M/S app) this could go away by migrating to High replication datastore. 


 I'm using the HR datastore already, but I'm not surprised with the 
 Timeouts as in my case there can sometimes, but rarely, be close to 10 
 transactions / seconds on the same entity group. 

  Hope this helps clarifies some points. 

 Thanks much for your detailed answers indeed! 

 - Pol 


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/5x_K_zD-N50J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



[google-appengine] Re: TransactionFailedError: Is the transaction committed or not?

2011-07-28 Thread thecheatah
I agree with Joshua. This kind of behavior would not be acceptable if
I was to make an app that requires 100% consistency and easy to
program. For stuff like this I use a relational db.

Ravneet

On Jul 28, 10:02 am, Joshua Smith joshuaesm...@charter.net wrote:
 The problem is that google transactions can report an exception, and then go 
 ahead and succeed anyway.

 So the docs recommend that you only write idempotent transactions, which is a 
 completely silly suggestion.  I've yet to see a single example of how one 
 might write an idempotent transaction.  (Unless, I suppose, you create a 
 separate child model in the database which is parented by the object you are 
 transacting on, and then you query the list of children every time you retry 
 your transaction to see if its already in there, but that won't scale.)

 I contend that a DB that cannot tell you reliably whether a transaction 
 succeeded for failed does not support transactions.

 GAE can essentially report 3 possible results from a transaction:
 - Definitely succeeded
 - Definitely failed
 - Beats me

 I contend that third possible result makes it impossible to write software 
 that relies on transactions.

 Therefore, GAE doesn't support transactions.

 On Jul 27, 2011, at 8:58 PM, Tim Hoffman wrote:







  If you always get modify put within a transaction how would it be 
  unreliable?

  Rgds

  Tim

  --
  You received this message because you are subscribed to the Google Groups 
  Google App Engine group.
  To view this discussion on the web 
  visithttps://groups.google.com/d/msg/google-appengine/-/mP_8kv_-LlMJ.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to 
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group 
  athttp://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



[google-appengine] Re: TransactionFailedError: Is the transaction committed or not?

2011-07-28 Thread Pol
On Jul 26, 9:40 pm, Jose Montes de Oca jfmontesde...@google.com
wrote:
 Hi Pol,

 What this meas is that even if a transaction throws an exception this does
 not means the transaction failed, thats why you need to make your datastore
 transaction idempotent. So if you retry a transaction because it throws an
 exception, your transaction needs to check if the last
 transaction committed successfully or not.

I don't understand how GAE could not report if the transaction
succeeded or not. Clearly this stuff is deterministic.

You're saying I have to maintain my own journal of transactions to
check against. But then that journal is itself would have to be
updated through transactions, so it needs to be journaled as well and
so on... it's completely unsolvable. How would implement a bank on
such a system? How could you ever 100% guarantee a user's transactions
are all in there or that the account's balance is correct?

Let's even look at a simple example: if you were to implement a simple
counter, how do you make that idempotent?

txn()
  counter = Counter.get(...)
  counter.value += 1
  counter.put()

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



[google-appengine] Re: TransactionFailedError: Is the transaction committed or not?

2011-07-28 Thread Pol
I see two big problems here:

1) There appear to be absolutely no record, say in the dashboard, of
failed transactions that eventually succeeded or really failed. For
some type of apps, I could live with 1 in a million transaction
failure that may or may not be eventually successful, but I need to
know which entity got in a degenerated state.

2) The benefit of the current approach of fake-real-transactions is
completely unknown: does it make GAE massively faster or more reliable
or something?

For instance, if you choose non-redundant storage in Amazon S3:
1) you get notifications for lost objects
2) it's cheaper

On Jul 28, 4:02 pm, Joshua Smith joshuaesm...@charter.net wrote:
 The problem is that google transactions can report an exception, and then go 
 ahead and succeed anyway.

 So the docs recommend that you only write idempotent transactions, which is a 
 completely silly suggestion.  I've yet to see a single example of how one 
 might write an idempotent transaction.  (Unless, I suppose, you create a 
 separate child model in the database which is parented by the object you are 
 transacting on, and then you query the list of children every time you retry 
 your transaction to see if its already in there, but that won't scale.)

 I contend that a DB that cannot tell you reliably whether a transaction 
 succeeded for failed does not support transactions.

 GAE can essentially report 3 possible results from a transaction:
 - Definitely succeeded
 - Definitely failed
 - Beats me

 I contend that third possible result makes it impossible to write software 
 that relies on transactions.

 Therefore, GAE doesn't support transactions.

 On Jul 27, 2011, at 8:58 PM, Tim Hoffman wrote:







  If you always get modify put within a transaction how would it be 
  unreliable?

  Rgds

  Tim

  --
  You received this message because you are subscribed to the Google Groups 
  Google App Engine group.
  To view this discussion on the web 
  visithttps://groups.google.com/d/msg/google-appengine/-/mP_8kv_-LlMJ.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to 
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group 
  athttp://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



Re: [google-appengine] Re: TransactionFailedError: Is the transaction committed or not?

2011-07-28 Thread Stephen Johnson
This does seem to be a conundrum. I'll offer up an idea for those instances
where this behavior is of concern. It's not a great solution but if you need
to guarantee that a transaction definitely succeeds and your transaction is
idempotent then maybe this or some variation will help.

First off, incrementing version numbers won't work due to the following
scenario:

1.) open a transaction find out the current version is 78
2.) perform update and increment the version to 79
3.) commit transaction
4.) get an exception. Now, did the transaction succeed or fail?? This is the
conundrum
5.) so re-fetch the item and check the version number, it says 79, so
transaction must have succeed, but what if some other transaction in the
meantime was the one that updated the item and incremented the version
number. So back to conundrum!!!

Okay, so instead of using a version number we use a random transaction stamp
(this could be a random number plus some hashed in data that is being
transacted, etc. to make it really unique). Then instead of a version number
property, we add a transaction stamp list that holds, for example, the last
5 or 10 stamps, or however many you want. So it goes like this:

1.) open a transaction
2.) perform update
3.) add unique transaction stamp to list, if list has reached maximum remove
the oldest one (FIFO).
4.) commit transaction
5.) if everything is fine, go on our merry way
6.) if exception is generated, then re-query the entity if the entity has
our unique timestamp in its list then the transaction really did succeed,
if not then the transaction really did fail.

Well, that's my idea,
Stephen
CortexConnect
cortexconnect.appspot.com


On Thu, Jul 28, 2011 at 7:51 AM, Pol i...@pol-online.net wrote:

 I see two big problems here:

 1) There appear to be absolutely no record, say in the dashboard, of
 failed transactions that eventually succeeded or really failed. For
 some type of apps, I could live with 1 in a million transaction
 failure that may or may not be eventually successful, but I need to
 know which entity got in a degenerated state.

 2) The benefit of the current approach of fake-real-transactions is
 completely unknown: does it make GAE massively faster or more reliable
 or something?

 For instance, if you choose non-redundant storage in Amazon S3:
 1) you get notifications for lost objects
 2) it's cheaper

 On Jul 28, 4:02 pm, Joshua Smith joshuaesm...@charter.net wrote:
  The problem is that google transactions can report an exception, and then
 go ahead and succeed anyway.
 
  So the docs recommend that you only write idempotent transactions, which
 is a completely silly suggestion.  I've yet to see a single example of how
 one might write an idempotent transaction.  (Unless, I suppose, you create a
 separate child model in the database which is parented by the object you are
 transacting on, and then you query the list of children every time you retry
 your transaction to see if its already in there, but that won't scale.)
 
  I contend that a DB that cannot tell you reliably whether a transaction
 succeeded for failed does not support transactions.
 
  GAE can essentially report 3 possible results from a transaction:
  - Definitely succeeded
  - Definitely failed
  - Beats me
 
  I contend that third possible result makes it impossible to write
 software that relies on transactions.
 
  Therefore, GAE doesn't support transactions.
 
  On Jul 27, 2011, at 8:58 PM, Tim Hoffman wrote:
 
 
 
 
 
 
 
   If you always get modify put within a transaction how would it be
 unreliable?
 
   Rgds
 
   Tim
 
   --
   You received this message because you are subscribed to the Google
 Groups Google App Engine group.
   To view this discussion on the web visithttps://
 groups.google.com/d/msg/google-appengine/-/mP_8kv_-LlMJ.
   To post to this group, send email to google-appengine@googlegroups.com
 .
   To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
   For more options, visit this group athttp://
 groups.google.com/group/google-appengine?hl=en.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



Re: [google-appengine] Re: TransactionFailedError: Is the transaction committed or not?

2011-07-28 Thread Stephen Johnson
Oops, I meant that your transaction is NOT idempotent, duh.

On Thu, Jul 28, 2011 at 10:31 AM, Stephen Johnson onepagewo...@gmail.comwrote:

 This does seem to be a conundrum. I'll offer up an idea for those instances
 where this behavior is of concern. It's not a great solution but if you need
 to guarantee that a transaction definitely succeeds and your transaction is
 idempotent then maybe this or some variation will help.

 First off, incrementing version numbers won't work due to the following
 scenario:

 1.) open a transaction find out the current version is 78
 2.) perform update and increment the version to 79
 3.) commit transaction
 4.) get an exception. Now, did the transaction succeed or fail?? This is
 the conundrum
 5.) so re-fetch the item and check the version number, it says 79, so
 transaction must have succeed, but what if some other transaction in the
 meantime was the one that updated the item and incremented the version
 number. So back to conundrum!!!

 Okay, so instead of using a version number we use a random transaction
 stamp (this could be a random number plus some hashed in data that is being
 transacted, etc. to make it really unique). Then instead of a version number
 property, we add a transaction stamp list that holds, for example, the last
 5 or 10 stamps, or however many you want. So it goes like this:

 1.) open a transaction
 2.) perform update
 3.) add unique transaction stamp to list, if list has reached maximum
 remove the oldest one (FIFO).
 4.) commit transaction
  5.) if everything is fine, go on our merry way
 6.) if exception is generated, then re-query the entity if the entity has
 our unique timestamp in its list then the transaction really did succeed,
 if not then the transaction really did fail.

 Well, that's my idea,
 Stephen
 CortexConnect
 cortexconnect.appspot.com


 On Thu, Jul 28, 2011 at 7:51 AM, Pol i...@pol-online.net wrote:

 I see two big problems here:

 1) There appear to be absolutely no record, say in the dashboard, of
 failed transactions that eventually succeeded or really failed. For
 some type of apps, I could live with 1 in a million transaction
 failure that may or may not be eventually successful, but I need to
 know which entity got in a degenerated state.

 2) The benefit of the current approach of fake-real-transactions is
 completely unknown: does it make GAE massively faster or more reliable
 or something?

 For instance, if you choose non-redundant storage in Amazon S3:
 1) you get notifications for lost objects
 2) it's cheaper

 On Jul 28, 4:02 pm, Joshua Smith joshuaesm...@charter.net wrote:
  The problem is that google transactions can report an exception, and
 then go ahead and succeed anyway.
 
  So the docs recommend that you only write idempotent transactions, which
 is a completely silly suggestion.  I've yet to see a single example of how
 one might write an idempotent transaction.  (Unless, I suppose, you create a
 separate child model in the database which is parented by the object you are
 transacting on, and then you query the list of children every time you retry
 your transaction to see if its already in there, but that won't scale.)
 
  I contend that a DB that cannot tell you reliably whether a transaction
 succeeded for failed does not support transactions.
 
  GAE can essentially report 3 possible results from a transaction:
  - Definitely succeeded
  - Definitely failed
  - Beats me
 
  I contend that third possible result makes it impossible to write
 software that relies on transactions.
 
  Therefore, GAE doesn't support transactions.
 
  On Jul 27, 2011, at 8:58 PM, Tim Hoffman wrote:
 
 
 
 
 
 
 
   If you always get modify put within a transaction how would it be
 unreliable?
 
   Rgds
 
   Tim
 
   --
   You received this message because you are subscribed to the Google
 Groups Google App Engine group.
   To view this discussion on the web visithttps://
 groups.google.com/d/msg/google-appengine/-/mP_8kv_-LlMJ.
   To post to this group, send email to
 google-appengine@googlegroups.com.
   To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
   For more options, visit this group athttp://
 groups.google.com/group/google-appengine?hl=en.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.


Re: [google-appengine] Re: TransactionFailedError: Is the transaction committed or not?

2011-07-28 Thread Robert Kluin
On Thu, Jul 28, 2011 at 12:31, Stephen Johnson onepagewo...@gmail.com wrote:
 This does seem to be a conundrum. I'll offer up an idea for those instances
 where this behavior is of concern. It's not a great solution but if you need
 to guarantee that a transaction definitely succeeds and your transaction is
 idempotent then maybe this or some variation will help.
 First off, incrementing version numbers won't work due to the following
 scenario:
 1.) open a transaction find out the current version is 78
 2.) perform update and increment the version to 79
 3.) commit transaction
 4.) get an exception. Now, did the transaction succeed or fail?? This is the
 conundrum
 5.) so re-fetch the item and check the version number, it says 79, so
 transaction must have succeed, but what if some other transaction in the
 meantime was the one that updated the item and incremented the version
 number. So back to conundrum!!!

This method doesn't really make sense if you read the version number
in the transaction, do stuff, increment the version, then reput within
the same transaction.

However, I do use something like this in some cases, except I have
read the current version prior to 'computing' my updated values.  So I
pass the expected version number into the transaction.  If I get a
mismatch then I know the update values might be wrong and the update
either needs skipped, recomputed, or the user needs to OK the changes
(and of course this happens based on the current version number).



 Okay, so instead of using a version number we use a random transaction stamp
 (this could be a random number plus some hashed in data that is being
 transacted, etc. to make it really unique). Then instead of a version number
 property, we add a transaction stamp list that holds, for example, the last
 5 or 10 stamps, or however many you want. So it goes like this:
 1.) open a transaction
 2.) perform update
 3.) add unique transaction stamp to list, if list has reached maximum remove
 the oldest one (FIFO).
 4.) commit transaction
 5.) if everything is fine, go on our merry way
 6.) if exception is generated, then re-query the entity if the entity has
 our unique timestamp in its list then the transaction really did succeed,
 if not then the transaction really did fail.

Again, the unique id needs generated outside the transaction so that
if the transaction were to run again the id would be the same.  Also,
instead of the list property you could use a separate entity (in the
same entity group) with no properties as a marker.  That avoids the
deserialization cost and is totally scalable.  With some minor tweaks
you would even be able to cleanup the old marker entities.

Also, as Stephen notes in a later comment, if the next read is in a
transaction it will force any outstanding writes to be applied -- so
it is always consistent.  Note that a db.get(some_key) implicitly uses
a transaction.


Robert



 Well, that's my idea,
 Stephen
 CortexConnect
 cortexconnect.appspot.com


 On Thu, Jul 28, 2011 at 7:51 AM, Pol i...@pol-online.net wrote:

 I see two big problems here:

 1) There appear to be absolutely no record, say in the dashboard, of
 failed transactions that eventually succeeded or really failed. For
 some type of apps, I could live with 1 in a million transaction
 failure that may or may not be eventually successful, but I need to
 know which entity got in a degenerated state.

 2) The benefit of the current approach of fake-real-transactions is
 completely unknown: does it make GAE massively faster or more reliable
 or something?

 For instance, if you choose non-redundant storage in Amazon S3:
 1) you get notifications for lost objects
 2) it's cheaper

 On Jul 28, 4:02 pm, Joshua Smith joshuaesm...@charter.net wrote:
  The problem is that google transactions can report an exception, and
  then go ahead and succeed anyway.
 
  So the docs recommend that you only write idempotent transactions, which
  is a completely silly suggestion.  I've yet to see a single example of how
  one might write an idempotent transaction.  (Unless, I suppose, you create 
  a
  separate child model in the database which is parented by the object you 
  are
  transacting on, and then you query the list of children every time you 
  retry
  your transaction to see if its already in there, but that won't scale.)
 
  I contend that a DB that cannot tell you reliably whether a transaction
  succeeded for failed does not support transactions.
 
  GAE can essentially report 3 possible results from a transaction:
  - Definitely succeeded
  - Definitely failed
  - Beats me
 
  I contend that third possible result makes it impossible to write
  software that relies on transactions.
 
  Therefore, GAE doesn't support transactions.
 
  On Jul 27, 2011, at 8:58 PM, Tim Hoffman wrote:
 
 
 
 
 
 
 
   If you always get modify put within a transaction how would it be
   unreliable?
 
   Rgds
 
   Tim
 
   --
   You received this message because you are 

[google-appengine] Re: TransactionFailedError: Is the transaction committed or not?

2011-07-28 Thread Jose Montes de Oca
Hi Pol,

Answering your first concern in numbers:

1) I think there is a misunderstanding on getting an exception for a 
failure. Because a transaction will succeed or fail. FYI we are aware this 
is unclear in the docs and we are working on it. But as I said before if you 
could live with 1 in a million transaction to not know fore sure the outcome 
(succeeded or failed) you wont need to make your process idempotent.

2) There is no fake-real-transaction, for example, one of the other uncommon 
exception that someone could get in the commit() would be Timeout exception. 
With any distributed system, unless you are willing to wait for an answer 
forever, there is always the possibility that you will get a timeout before 
the remote system responds with the result of the operation you requested 
(this is true in all databases), what GAE warranty is that the transaction 
either commits successfully or it does not, so if you implement some retry 
logic in transactions, having it idempotent will warranty you don't do the 
transaction twice.

For the counter example, to make something idempotent you need a way to 
distinguish each time the counter gets incremented, it would depend on the 
logic of the application. its really hard to make it idempotent by itself. 
lets say part of a transaction would be incrementing a counter, if the 
process of incrementing is unique to users and a user can only incremented 
once, you could maintain another kind that gets set inside the transaction 
that stores the user that incremented the counter. then your logic inside 
the transaction will first check if the user already incremented the counter 
by making a get to the kind that keeps track of that (we can assure that if 
a user appears here the transaction was committed because a transaction 
either fails or not) if he did not, you go ahead and process the logic 
again, else do nothing (it already did it). This will make sure the 
transaction is not redone if you retry it after getting an exception where 
the outcome was unknown.

But as I said before, this exceptions are uncommon and it would be good to 
know what are you doing inside the transaction that is causing those 
transactions. Timeout exception could be related to master slaves issues (in 
a M/S app) this could go away by migrating to High replication datastore.

Hope this helps clarifies some points.

Best,
Jose

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/klKa70GFvwAJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.