I'm curious what consensus looks like. In what forum among which stakeholders. Clearly among developers who have some knowledge of Djangos innards, and so I suspect here. But I find conversation here on this thread so neatly short I can digest it in a short read, and
https://code.djangoproject.com/ticket/24731 https://code.djangoproject.com/ticket/12938 both touch on the same issue with similarly delectably short comment streams that I could read and digest them in a jiffy. Read another way, not a lot of consensus generation discussion or activity visible. So I'll play the newb (because I am I guess, well been coding with Django for almost 4 years very very casually part time, and have had this particular issue on my "to solve" list for a long time already. But for me to make progress on it I need to invest some time into learning and some of that empirically through experimentation but I'll try and stimulate a little conversation here by doing some of it through the posing of a possibly naive question or two. Before that let me state that in my application (and I suspect this is not unusual, I am trying to validate m2m relations and one2m as well in what I'd describe as a rich model object, I use the term rich to describe a handful of models related to one another, all of which are updated with one form submission. In my simplest example I am modelling a game session which has a one2many relationship to ranks, which have a one2one relationship to teams which have an m2m relationship to players. A single game session is logged as the ranked list of teams and their players ... but I wouldn't get lost in the one example, the point is such relationships can't be wildly uncommon and we would all like to keep the validation logic in the models. In my example a session would like to check that there are the right number of ranks, not too few, not too many, as the game allows. Validating this in the form is frustrating as it's a) somewhat more complicated and b) not as secure (allows erroneous saves through means other than this form). And the logic belongs in the model. The session knows how many ranks it's expecting. I've seen other examples just as clear. And so onto the learning through possibly naive questioning. As I see it there are two versions of this rich object (objects of several models all interrelated: 1) The database version 2) the ORM version The problem I see is that the ORM versions lack primary keys and the relationships they create until they are saved. So aside from inviting correction of any misunderstandings I may have tabled I will ask the salient question: Is this not a classic application for transactions. Namely we save all of the objects without committing then do the validation of the relationships in the ORM. This seems naive to me as it presupposes a few things that may or may not be true (and I fear are not): 1) The an uncommitted transaction delivers primary keys 2) That an uncommitted transaction can easily be reflected back in the ORM If these are possible, is it not a good chance then to validate the relations int he respective model's Clean() methods and through an exception on failure, that results in a complete roll back and if it succeeds results in a commit. I invite commentary, and discussion in the hope of achieving the elusive beast of consensus that Frederico alludes to. The main need I see is to create the relationships in the ORM, and it may be possible to do this pre-save too, with place-holder PKs, I do exactly this sort of thing at the form level, managing the relations between the various form elements, with placeholder IDs on elements. And so I imagine loosely if the strategy above is impossible that another might implement something along these lines, a way to create all the objects and have them as ORM objects but with placeholder PKs where needed managing the relations and (a central difficult) a modelled through object for M2Ms - which I imagine can be modelled as a django model with a ForeignKey to each of the related models one such object capturing the relationshiips. Where real PKs exist, they can be used, where they do not exist a placeholder could be used to model relationships in the ORM only. On save these placeholders are ignored and the database creates PKs as usual. I can imagine a number of ways of modelling such placeholders. Either simply encoding otherwise illegal values (-ve numbers), or reserving on ID to flag a placheholder (PK-0 or PK=Maxvalue) and the placeholder PK (PH) is another attribute on model with references to PK returning that if the reserved value is in place etc. Alternately just use legal values in the high range (max value down) for the PKs and use a state flag to indicate they are faux, and to be ignored on save (it could be the simple existing flag that tells us it's unsaved can serve that role). Anyhow a pile of naive questions and speculations and I hope it stimulates a response or two or three and something that heads towards consensus. Even if it's something completely different and all this is dismissed ;-). Kind regards, Bernd. On Saturday, 5 December 2015 04:59:05 UTC+11, Federico Capoano wrote: > > It could be a potential ticket to work on my next django dev sprint. > > But first it would be nice to have some basic consensus on how to proceed. > > Was it ever discussed in any older thread or ticket? > > > > On Thursday, December 3, 2015 at 5:21:06 PM UTC+1, Tim Graham wrote: >> >> Here's an open ticket about adding model level validation for >> many-to-many: https://code.djangoproject.com/ticket/12938 >> >> On Thursday, December 3, 2015 at 11:04:22 AM UTC-5, Federico Capoano >> wrote: >>> >>> Thanks Aymeric, >>> >>> I decided to post this problem on django-developers because I've read >>> this ticket: >>> https://code.djangoproject.com/ticket/24731 >>> Sorry for omitting this information. >>> >>> Has there been a discussion about this topic already? >>> >>> Would it be hard to implement an easier solution into django? >>> >>> I spent a few hours working on this issue, I consider myself fluent with >>> django and it's quite shocking I had to put such an amount of effort just >>> to validate many2many relationships before they are saved to the database. >>> >>> IMHO it would be better if we could do one of these two (or even both) >>> things: >>> >>> 1. make this process easier in future django versions >>> 2. document the current best practice to solve this problem in current >>> django to save people's time >>> >>> What do people you think? >>> >>> Here's my solution working with django 1.9: >>> https://github.com/openwisp/django-netjsonconfig/compare/4082988...master >>> >>> What do you think of it? Can it be improved in some way? >>> >>> Federico >>> >>> >>> On Thursday, December 3, 2015 at 1:43:21 PM UTC+1, Aymeric Augustin >>> wrote: >>>> >>>> Hello Frederico, >>>> >>>> It appears that you're hitting the problem described in the "Avoid >>>> catching exceptions inside atomic!" warning on this page: >>>> >>>> https://docs.djangoproject.com/en/1.8/topics/db/transactions/#handling-exceptions-within-postgresql-transactions >>>> >>>> To obtain that sort of result, I suppose you must be catching an >>>> IntegrityError, re-raising a ValidationError, which Django in turn >>>> catches, >>>> and then you hit that traceback. >>>> >>>> Adding an atomic block inside your try/catch block that catches the >>>> IntegrityError will resolve that particular problem — putting that part of >>>> the discussion into django-users territory. >>>> >>>> If this isn't what happens, please post your code and ask for help on >>>> django-users. >>>> >>>> -- >>>> Aymeric >>>> >>>> 2015-12-03 13:28 GMT+01:00 Federico Capoano <[email protected]>: >>>> >>>>> Hi everybody, >>>>> >>>>> I am sure it has happened to many of you. >>>>> >>>>> Validating m2m BEFORE saving the relationships is very hard and time >>>>> consuming. >>>>> >>>>> Now this solution: >>>>> http://schinckel.net/2012/02/06/pre-validating-many-to-many-fields./ >>>>> >>>>> Proposes to solve it with a ModelForm in the admin. >>>>> Cool, that works. >>>>> >>>>> But, if I want to enforce validation on the model, to avoid corrupted >>>>> data, I notice the signal is executed in a transaction block, in which if >>>>> I >>>>> raise a ValidationError I get the following: >>>>> >>>>> Traceback (most recent call last): >>>>> File >>>>> "/var/www/django-netjsonconfig/django_netjsonconfig/tests/test_device.py", >>>>> >>>>> line 106, in test_m2m_validation >>>>> d.templates.add(t) >>>>> File >>>>> "/home/nemesis/.virtualenvs/djnetconfig3/lib/python3.4/site-packages/django/db/models/fields/related_descriptors.py", >>>>> >>>>> line 843, in add >>>>> self._add_items(self.source_field_name, self.target_field_name, >>>>> *objs) >>>>> File >>>>> "/home/nemesis/.virtualenvs/djnetconfig3/lib/python3.4/site-packages/sortedm2m/fields.py", >>>>> >>>>> line 138, in _add_items >>>>> for val in vals: >>>>> File >>>>> "/home/nemesis/.virtualenvs/djnetconfig3/lib/python3.4/site-packages/django/db/models/query.py", >>>>> >>>>> line 258, in __iter__ >>>>> self._fetch_all() >>>>> File >>>>> "/home/nemesis/.virtualenvs/djnetconfig3/lib/python3.4/site-packages/django/db/models/query.py", >>>>> >>>>> line 1074, in _fetch_all >>>>> self._result_cache = list(self.iterator()) >>>>> File >>>>> "/home/nemesis/.virtualenvs/djnetconfig3/lib/python3.4/site-packages/django/db/models/query.py", >>>>> >>>>> line 158, in __iter__ >>>>> for row in compiler.results_iter(): >>>>> File >>>>> "/home/nemesis/.virtualenvs/djnetconfig3/lib/python3.4/site-packages/django/db/models/sql/compiler.py", >>>>> >>>>> line 806, in results_iter >>>>> results = self.execute_sql(MULTI) >>>>> File >>>>> "/home/nemesis/.virtualenvs/djnetconfig3/lib/python3.4/site-packages/django/db/models/sql/compiler.py", >>>>> >>>>> line 852, in execute_sql >>>>> cursor.execute(sql, params) >>>>> File >>>>> "/home/nemesis/.virtualenvs/djnetconfig3/lib/python3.4/site-packages/django/db/backends/utils.py", >>>>> >>>>> line 59, in execute >>>>> self.db.validate_no_broken_transaction() >>>>> File >>>>> "/home/nemesis/.virtualenvs/djnetconfig3/lib/python3.4/site-packages/django/db/backends/base/base.py", >>>>> >>>>> line 429, in validate_no_broken_transaction >>>>> "An error occurred in the current transaction. You can't " >>>>> django.db.transaction.TransactionManagementError: An error occurred in >>>>> the current transaction. You can't execute queries until the end of the >>>>> 'atomic' block. >>>>> >>>>> This is surely an area that needs improvement in django. >>>>> >>>>> Why is it so hard? >>>>> >>>>> Best regards >>>>> Federico >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Django developers (Contributions to Django itself)" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To post to this group, send email to [email protected]. >>>>> Visit this group at http://groups.google.com/group/django-developers. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/django-developers/2e6e82d0-0645-4fd7-8905-d327c99b6352%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/django-developers/2e6e82d0-0645-4fd7-8905-d327c99b6352%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> >>>> >>>> -- >>>> Aymeric. >>>> >>> -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/89c2bda2-d7a8-45bf-b215-69f01b442b45%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
