Re: Django Transaction Integrity

[Piet Delport]

Ian, thanks for the feedback, but what your concern is not related to
this proposal.

What you're talking about is issuing a BEGIN while already in a database
transaction, which is normally an error, with no standard meaning or
definition in SQL's transaction model.  It's completely up to the
database what happens:  MySQL will implicitly commit and start a new
transaction, as you point out, but (for example) PostgreSQL will just
issue a warning and ignore it, and SQLite will fail with an error.
Django never tries to do this, and never should.

What the original message talks about is how transactional blocks marked
in Python translate to SQL transactions.  This is something every SQL
abstraction layer does somewhere:  the canonical way is to associate the
execution of each transactional block with a corresponding SQL
transaction, and (because execution nests but SQL transactions cannot)
most frameworks support nested transactional blocks by subsuming them,
associating their execution with the context's existing SQL
transaction.[1]  This is what Django currently does, modulo the bugs
described in my original message.

The proposal is to fix these bugs, making Django fully and reliably
transactional.  There are no database portability implications: Django's
transaction management will continue to work with any backend as before.

[1] This is sound because inner extents retain transactional (ACID)
semantics as part of the outer extent: no integrity is lost.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Django Transaction Integrity

[Ian Lewis]

> Yes: we're talking about Django's transaction management layer (that is, the
> code in django.db.transaction), which controls how Python-level transaction
> blocks map to SQL transactions.
>
> The original email and writeup has a full explanation. :)
>

Yes. What I was getting at is that django's transaction management
layer is a layer over top of the database's own transaction management
not a full transaction layer implemented at the application level. In
any case the underlying database doesn't support nested transactions
in the case of MySQL. Starting a transaction within a transaction does
not subsume the current transaction. It does an implicit commit,
ending the current transaction and starts a new top level transaction.

In any case it's probably not worth talking much more about it since
you can probably implement the functionality using savepoints at least
with InnoDB. But I did want to bring up that there are database
portability implications here.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Django Transaction Integrity

[Piet Delport]

2010/5/18 Ian Lewis :
> Am I missing something? Are we talking about doing some kind of
> transaction management at the application level?

Yes: we're talking about Django's transaction management layer (that is, the
code in django.db.transaction), which controls how Python-level transaction
blocks map to SQL transactions.

The original email and writeup has a full explanation. :)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Django Transaction Integrity

[Ian Lewis]

I should probably be more specific. Without using savepoints nested
transaction functionality couldn't be achieved with MySQL (and only
with InnoDB). START TRANSACTION does an implicit commit so the
previous transaction is not subsumed.

I suppose that savepoint based functionality could be implemented for
MySQL in lieu of nested transactions though.

See: http://dev.mysql.com/doc/refman/5.0/en/implicit-commit.html

2010/5/18 Ian Lewis :
> Am I missing something? Are we talking about doing some kind of
> transaction management at the application level? If not then it's
> worth noting that in MySQL you cannot begin a transaction twice
> without commiting or rolling back. i.e. nested transactions are not
> supported.
>
> On Tue, May 18, 2010 at 1:05 AM, Piet Delport  wrote:
>> 2010/5/17 Ian Lewis :
>>>
>>> I suppose that it's worth mentioning that any support for nested
>>> transactions will not be portable to different databases (particularly
>>> MySQL) and probably would need to be documented accordingly.
>>
>> How do you mean?
>>
>> Django-level transaction block nesting is portable to any database that
>> supports transactions: The top-level transaction subsumes nested
>> transactional blocks.
>>
>> If you are referring to SQL savepoints, supporting them is entirely
>> optional: nested transaction blocks do not depend on it in any way.
>> Savepoints are more like an optional performance optimization than
>> anything else, and just give applications the option of doing
>> finer-grained failure recovery in certain cases without changing the
>> meaning of transactional blocks. (The proposal above only mentions
>> savepoints for completeness, and because support for them would be
>> natural to integrate.)
>>
>> --
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers" group.
>> To post to this group, send email to django-develop...@googlegroups.com.
>> To unsubscribe from this group, send email to 
>> django-developers+unsubscr...@googlegroups.com.
>> For more options, visit this group at 
>> http://groups.google.com/group/django-developers?hl=en.
>>
>>
>
>
>
> --
> ===
> 株式会社ビープラウド  イアン・ルイス
> 〒150-0021
> 東京都渋谷区恵比寿西2-3-2　NSビル6階
> email: ianmle...@beproud.jp
> TEL:03-6416-9836
> FAX:03-6416-9837
> http://www.beproud.jp/
> ===
>



-- 
===
株式会社ビープラウド  イアン・ルイス
〒150-0021
東京都渋谷区恵比寿西2-3-2　NSビル6階
email: ianmle...@beproud.jp
TEL:03-6416-9836
FAX:03-6416-9837
http://www.beproud.jp/
===

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Django Transaction Integrity

[Ian Lewis]

Am I missing something? Are we talking about doing some kind of
transaction management at the application level? If not then it's
worth noting that in MySQL you cannot begin a transaction twice
without commiting or rolling back. i.e. nested transactions are not
supported.

On Tue, May 18, 2010 at 1:05 AM, Piet Delport  wrote:
> 2010/5/17 Ian Lewis :
>>
>> I suppose that it's worth mentioning that any support for nested
>> transactions will not be portable to different databases (particularly
>> MySQL) and probably would need to be documented accordingly.
>
> How do you mean?
>
> Django-level transaction block nesting is portable to any database that
> supports transactions: The top-level transaction subsumes nested
> transactional blocks.
>
> If you are referring to SQL savepoints, supporting them is entirely
> optional: nested transaction blocks do not depend on it in any way.
> Savepoints are more like an optional performance optimization than
> anything else, and just give applications the option of doing
> finer-grained failure recovery in certain cases without changing the
> meaning of transactional blocks. (The proposal above only mentions
> savepoints for completeness, and because support for them would be
> natural to integrate.)
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Django developers" group.
> To post to this group, send email to django-develop...@googlegroups.com.
> To unsubscribe from this group, send email to 
> django-developers+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/django-developers?hl=en.
>
>



-- 
===
株式会社ビープラウド  イアン・ルイス
〒150-0021
東京都渋谷区恵比寿西2-3-2　NSビル6階
email: ianmle...@beproud.jp
TEL:03-6416-9836
FAX:03-6416-9837
http://www.beproud.jp/
===

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Django Transaction Integrity

[Piet Delport]

2010/5/17 Ian Lewis :
>
> I suppose that it's worth mentioning that any support for nested
> transactions will not be portable to different databases (particularly
> MySQL) and probably would need to be documented accordingly.

How do you mean?

Django-level transaction block nesting is portable to any database that
supports transactions: The top-level transaction subsumes nested
transactional blocks.

If you are referring to SQL savepoints, supporting them is entirely
optional: nested transaction blocks do not depend on it in any way.
Savepoints are more like an optional performance optimization than
anything else, and just give applications the option of doing
finer-grained failure recovery in certain cases without changing the
meaning of transactional blocks. (The proposal above only mentions
savepoints for completeness, and because support for them would be
natural to integrate.)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Django Transaction Integrity

[Ian Lewis]

I suppose that it's worth mentioning that any support for nested
transactions will not be portable to different databases (particularly
MySQL) and probably would need to be documented accordingly.

Ian

On Mon, May 17, 2010 at 7:50 PM, Piet Delport  wrote:
> Hi Russel, thanks for the response!
>
> On May 15, 12:47 pm, Russell Keith-Magee 
> wrote:
>>
>> By way of a brief explanation of why things are they way they are:
>>
>> Firstly, Django's history in journalism-based websites breeds a
>> history of high-volume reads, and low-volume, relatively simple
>> writes. As a result, the transactional requirements have been fairly
>> light.
>
> I suspected as much; the current implementation doesn't make much
> sense
> unless managed blocks were informally introduced at some point without
> being relied upon to be transactional. While this makes managed blocks
> are a bit of a white elephant at the moment (the only code that can
> use
> them safely is code that doesn't need to be transactional), it might
> also be a minor blessing in disguise, in that they are not used widely
> enough for the problem to have major impact (yet).
>
>> Secondly, although it's entirely legal Python to apply
>> commit_on_success et al on arbitrary functions, the most common usage
>> has been to apply them to view functions, not arbitrary processing
>> functions. When these decorators are applied at the view level, you
>> don't get the nesting problems you describe.
>
> Right.
>
> (I'd like to note, though, that limiting commit_on_success to only
> top-level view functions is not enough to completely avoid this bug:
> for
> example, any usage of TransactionMiddleware will interact badly with
> it,
> especially if any intervening middleware uses the database.)
>
>> I haven't completely absorbed the specific proposals from your
>> document, but to put something into your mind for consideration:
>> backwards compatibility is a very large consideration here. We can't
>> arbitrarily change the way that transactions operate in existing code.
>> Any change that we make must be by way of adding new
>> functions/decorators, or by adding flags/modifiers to the existing
>> functions and decorators with default values that reproduce existing
>> behavior.
>
> Right, backward compatibility was big concern for me while
> investigating
> solutions to this. I believe that the proposed fix (making transaction
> blocks transactional as advertised) can be implemented in such a way
> that backward compatibility is preserved for almost all users: user
> code
> would not need to know anything about this change except that their
> currently-declared transaction blocks will operate correctly as
> intended, and no longer silently be vulnerable to broken edge cases as
> described above.
>
> In other words, the proposal is not to significantly change the
> meaning
> of transaction blocks, but rather to make them operate fully (instead
> of
> partially) as advertised: all code that currently uses them more or
> less
> as expected should continue to work as before, only more safely and
> correctly.
>
> Of course, this will need thorough review and testing, in case there
> turns out to be real cases where user code has somehow come to depend
> on
> the broken edge cases, but i can't offhandedly think of a scenario
> where
> this would easily happen.
>
>> So - thanks for your contribution. If you could kindly keep this warm
>> for a couple of weeks, we can revisit this properly once Django 1.3
>> development is in full swing. In the interim -- anything you can do to
>> turn this abstract set of proposals into some sample code, test cases,
>> or prototype implementation is certainly most welcome, and the more
>> concrete evidence you can present, the easier it will be to turn this
>> into a change in trunk.
>
> Thank you for your advice; this sounds like a good plan!
>
> I have informal tests for the scenarios described in the writeup, and
> will begin readying a prototype implementation as time allows.
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Django developers" group.
> To post to this group, send email to django-develop...@googlegroups.com.
> To unsubscribe from this group, send email to 
> django-developers+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/django-developers?hl=en.
>
>



-- 
===
株式会社ビープラウド  イアン・ルイス
〒150-0021
東京都渋谷区恵比寿西2-3-2　NSビル6階
email: ianmle...@beproud.jp
TEL:03-6416-9836
FAX:03-6416-9837
http://www.beproud.jp/
===

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en

Re: Django Transaction Integrity

[Piet Delport]

Hi Russel, thanks for the response!

On May 15, 12:47 pm, Russell Keith-Magee 
wrote:
>
> By way of a brief explanation of why things are they way they are:
>
> Firstly, Django's history in journalism-based websites breeds a
> history of high-volume reads, and low-volume, relatively simple
> writes. As a result, the transactional requirements have been fairly
> light.

I suspected as much; the current implementation doesn't make much
sense
unless managed blocks were informally introduced at some point without
being relied upon to be transactional. While this makes managed blocks
are a bit of a white elephant at the moment (the only code that can
use
them safely is code that doesn't need to be transactional), it might
also be a minor blessing in disguise, in that they are not used widely
enough for the problem to have major impact (yet).

> Secondly, although it's entirely legal Python to apply
> commit_on_success et al on arbitrary functions, the most common usage
> has been to apply them to view functions, not arbitrary processing
> functions. When these decorators are applied at the view level, you
> don't get the nesting problems you describe.

Right.

(I'd like to note, though, that limiting commit_on_success to only
top-level view functions is not enough to completely avoid this bug:
for
example, any usage of TransactionMiddleware will interact badly with
it,
especially if any intervening middleware uses the database.)

> I haven't completely absorbed the specific proposals from your
> document, but to put something into your mind for consideration:
> backwards compatibility is a very large consideration here. We can't
> arbitrarily change the way that transactions operate in existing code.
> Any change that we make must be by way of adding new
> functions/decorators, or by adding flags/modifiers to the existing
> functions and decorators with default values that reproduce existing
> behavior.

Right, backward compatibility was big concern for me while
investigating
solutions to this. I believe that the proposed fix (making transaction
blocks transactional as advertised) can be implemented in such a way
that backward compatibility is preserved for almost all users: user
code
would not need to know anything about this change except that their
currently-declared transaction blocks will operate correctly as
intended, and no longer silently be vulnerable to broken edge cases as
described above.

In other words, the proposal is not to significantly change the
meaning
of transaction blocks, but rather to make them operate fully (instead
of
partially) as advertised: all code that currently uses them more or
less
as expected should continue to work as before, only more safely and
correctly.

Of course, this will need thorough review and testing, in case there
turns out to be real cases where user code has somehow come to depend
on
the broken edge cases, but i can't offhandedly think of a scenario
where
this would easily happen.

> So - thanks for your contribution. If you could kindly keep this warm
> for a couple of weeks, we can revisit this properly once Django 1.3
> development is in full swing. In the interim -- anything you can do to
> turn this abstract set of proposals into some sample code, test cases,
> or prototype implementation is certainly most welcome, and the more
> concrete evidence you can present, the easier it will be to turn this
> into a change in trunk.

Thank you for your advice; this sounds like a good plan!

I have informal tests for the scenarios described in the writeup, and
will begin readying a prototype implementation as time allows.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Django Transaction Integrity

[Andrea Zilio]

Hi Piet,

I'm just a django user, but I like very much your clear explanation
about how django transactions works and what's wrong with them!

Great work!

Andrea

On 13 Mag, 17:01, Piet Delport  wrote:
> Hi,
>
> I'm working with a company that uses and maintains a commercial Django-based
> commercial system, which requires reliable transactional integrity across
> groups of database operations in a number of places (financial transfers,
> and so on).
>
> I've been investigating how Django's transaction management works, but found
> to my surprise that Django's transaction managed blocks do not actually
> ensure any transactional integrity, despite the documentation indicating
> otherwise. (For example, "Use the commit_on_success decorator to use a
> single transaction for all the work done in a function.")
>
> I wrote up the following document, which describes the problem in a fair
> amount of detail, as well as how Django's transaction management system may
> be fixed to avoid it:
>
>    - Django Transaction
> Integrity
>
> I'd like to use this as a starting point for discussion; specifically, i
> want to solicit feedback about the proposed solution approach (subsuming
> nested transaction blocks): Does it sound reasonable to change
> django.db.transaction's implementation to this?
>
> I know this is not a small change, but i think it should be worth it if
> transaction blocks are to be supported at all: there seems to be very little
> point in keeping the current implementation where declaring a transaction
> block doesn't actually make it transactional.
>
> (This probably affects a number of current Django tickets, including at
> least: #2227 ,
> #6669
> , #9964 )
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Django developers" group.
> To post to this group, send email to django-develop...@googlegroups.com.
> To unsubscribe from this group, send email to 
> django-developers+unsubscr...@googlegroups.com.
> For more options, visit this group 
> athttp://groups.google.com/group/django-developers?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Django Transaction Integrity

[Russell Keith-Magee]

On Thu, May 13, 2010 at 11:01 PM, Piet Delport  wrote:
> Hi,
> I'm working with a company that uses and maintains a commercial Django-based
> commercial system, which requires reliable transactional integrity across
> groups of database operations in a number of places (financial transfers,
> and so on).
> I've been investigating how Django's transaction management works, but found
> to my surprise that Django's transaction managed blocks do not actually
> ensure any transactional integrity, despite the documentation indicating
> otherwise. (For example, "Use the commit_on_success decorator to use a
> single transaction for all the work done in a function.")
> I wrote up the following document, which describes the problem in a fair
> amount of detail, as well as how Django's transaction management system may
> be fixed to avoid it:
>
> Django Transaction Integrity
>
> I'd like to use this as a starting point for discussion; specifically, i
> want to solicit feedback about the proposed solution approach (subsuming
> nested transaction blocks): Does it sound reasonable to change
> django.db.transaction's implementation to this?
> I know this is not a small change, but i think it should be worth it if
> transaction blocks are to be supported at all: there seems to be very little
> point in keeping the current implementation where declaring a transaction
> block doesn't actually make it transactional.
> (This probably affects a number of current Django tickets, including at
> least: #2227, #6669, #9964)

Hi Piet,

You're certainly given us a lot to ruminate on here!

I haven't had a chance to fully digest what you've written, but from
my first inspection, you appear to have identified an area where
Django has room for improvement.

By way of a brief explanation of why things are they way they are:

Firstly, Django's history in journalism-based websites breeds a
history of high-volume reads, and low-volume, relatively simple
writes. As a result, the transactional requirements have been fairly
light.

Secondly, although it's entirely legal Python to apply
commit_on_success et al on arbitrary functions, the most common usage
has been to apply them to view functions, not arbitrary processing
functions. When these decorators are applied at the view level, you
don't get the nesting problems you describe.

The transaction functions as currently implemented are born of these
two factors, combined with the "deadlines" part of "perfectionists
with deadlines". In this context, Django's transactional integrity is
fine - it's only in the complex cases where problems start to emerge.

I'm certainly interested in addressing these complex cases, though.

So what's the way forward? Procedurally - we're in the final throes of
the release of 1.2, after which I will collapse for a little bit, then
get on a plane to travel to DjangoCon.eu. In practical terms, this
means that regardless of the merit of this discussion, not much is
going to happen for about 2 weeks.

I haven't completely absorbed the specific proposals from your
document, but to put something into your mind for consideration:
backwards compatibility is a very large consideration here. We can't
arbitrarily change the way that transactions operate in existing code.
Any change that we make must be by way of adding new
functions/decorators, or by adding flags/modifiers to the existing
functions and decorators with default values that reproduce existing
behavior.

So - thanks for your contribution. If you could kindly keep this warm
for a couple of weeks, we can revisit this properly once Django 1.3
development is in full swing. In the interim -- anything you can do to
turn this abstract set of proposals into some sample code, test cases,
or prototype implementation is certainly most welcome, and the more
concrete evidence you can present, the easier it will be to turn this
into a change in trunk.

Yours,
Russ Magee %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.