Should saving a model with foreign keys referencing unsaved models raise an error?

[Yo-Yo Ma]

The following example can throw a wrench in things, if you don't catch it 
right away, since it fails silently.

>>> instance.some_fk_field = unsaved_instance
>>> instance.save()

The following example bit somebody I worked with a couple years back as 
well.

>>> instance.some_m2m.add(unsaved_instance)

Would it suffice to modify the ORM to fail loudly when either of the above 
examples occur, or do they both represent documented features (thus 
preventing them from being "fixed")?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[GSoC 2013] Composite fields: first draft of the proposal

[Michal Petrucha]

Hello,

Last week I posted a draft of a timeline for my proposed project and
only afterward I realized that the timeline by itself does not convey
any meaningful information without explaining each item a little bit
deeper. Here comes the full version which I managed to finally
complete a while ago.

Again, for those who prefer reading the formatted version, the
proposal is also available as a gist:
https://gist.github.com/konk/5408673

Apologies for the huge amounts of text, I seem to have lost track
while writing.

Have an awesome day
Michal



Composite Fields, vol. 2


Abstract


Two years ago, as part of GSoC 2011, I started working on an
implementation of composite fields as a means to support multi-column
primary keys in models. While the project was was successful, the
timeframe wasn't sufficient to finish it and get it into a state where it
could be merged into Django. This year I propose to take the last
remaining steps for this project to be ready and hopefully even extend it
with some extra features which were left out of the initial project.

Aim of this project
===

Since I started working on this two years ago, I managed (with the help of
a few people) to get most of the hard work done. The biggest part that I
didn't get around to implementing was support in the ORM for multi-column
joins. This has, however, been implemented recently by Jeremy Tillman and
Anssi, which means there are only a few things left to be done.

First of all, the code sitting in my github repo is badly out of date,
which means it needs to be updated to the current state of master. While
I'm at it, I also want to clean up the revision history to get it as close
to a state where it could be just reviewed and merged directly as
possible.

Beside this, there are only the juicier features left that we initially
wanted to leave unsupported and get back to them later. Those are the
following (in no particular order):

- ``GenericForeignKey`` support for ``CompositeField``
- more intelligent handling of ``__in`` lookups
- database instrospection and ``inspectdb``
- detection of a change of the primary key in model instances

Summary of ``CompositeField``
=

This section summarizes the basic API as established in the proposal for
GSoC 2011 [1]_.

A ``CompositeField`` requires a list of enclosed regular model fields as
positional arguments, as shown in this example::

class SomeModel(models.Model):
first_field = models.IntegerField()
second_field = models.CharField(max_length=100)
composite = models.CompositeField(first_field, second_field)

The model class then contains a descriptor for the composite field, which
returns a ``CompositeValue`` which is a customized namedtuple, the
descriptor accepts any iterable of the appropriate length. An example
interactive session::

>>> instance = new SomeModel(first_field=47, second_field="some string")
>>> instance.composite
CompositeObject(first_field=47, second_field='some string')
>>> instance.composite.first_field
47
>>> instance.composite[1]
'some string'
>>> instance.composite = (74, "other string")
>>> instance.first_field, instance.second_field
(74, 'other string')

``CompositeField`` supports the following standard field options:
``unique``, ``db_index``, ``primary_key``. The first two will simply add a
corresponding tuple to ``model._meta.unique_together`` or
``model._meta.index_together``. Other field options don't make much sense
in the context of composite fields.

Supported ``QuerySet`` filters will be ``exact`` and ``in``. The former
should be clear enough, the latter is elaborated in a separate section.

It will be possible to use a ``CompositeField`` as a target field of
``ForeignKey``, ``OneToOneField`` and ``ManyToManyField``. This is
described in more detail in the following section.

Changes in ``ForeignKey``
=

Currently ``ForeignKey`` is a regular concrete field which manages both
the raw value stored in the database and the higher-level relationship
semantics. Managing the raw value is simple enough for simple
(single-column) targets. However, in the case of a composite target field,
this task becomes more complex. The biggest problem is that many parts of
the ORM work under the assumption that for each database column there is a
model field it can assign the value from the column to. While it might be
possible to lift this restriction, it would be a really complex project by
itself.

On the other hand, there is the abstraction of virtual fields working on
top of other fields which is required for this project anyway. The way
forward would be to use this abstraction for relationship fields.
Currently, ``ForeignKey`` (and by extension ``OneToOneField``) is the only
field whose ``name`` and ``attname`` differ, where ``name`` stores the
value dictated by the semantics of the field and ``attname`` 

Re: [GSoC 2013] Improved error reporting

[Russell Keith-Magee]

On Thu, Apr 25, 2013 at 3:00 AM, Andrew Godwin  wrote:

> Hi Damian,
>
> We don't generally accept GSOC projects that are just a bit grab bag of
> problems - this is looking a little bit like that. I'd like to see a better
> breakdown of what kind of time each ticket would take and what your
> planning schedule would be - in particular, I'd like to make sure you have
> roughly 12 full weeks worth of work, as some of these tickets are quite
> easy to fix.
>
> It would also be nice to see you pick fewer issues but ones which were
> more difficult - a past security GSOC just focused on three or four main,
> difficult problems, and while not all got done, it developed in a
> reasonable way.
>

I agree with Andrew - we're not just looking for "fix a bunch of bugs" as a
GSoC project. Don't get me wrong - bug fixes are good, but you're in a
unique position of having a 12 week dedicated period for working on Django.
This is an opportunity to fix a problems at a deep level, not just skim
over a lot of surface.

However, your ticket triage isn't necessarily all a waste. An alternative
approach to your proposal would be to identify broad or systematic
problems, and use the tickets as exemplars of how that problem manifests.

An example of this (that doesn't come from GSoC) was Ben Firshman's
handling of syndication back in the days of Django 1.1. Ben identified that
there were a lot of open bugs in the syndication framework, and he analysed
the trends and causes of those bugs -- for the most part, IIRC, subtle
variations between Atom and RSS formats. He then identified that the
structure of the existing code was the cause of these problems, and
proposed an alternative structure that would address these problems - and
probably many more related problems.

So - instead of just proposing a bunch of tickets to fix, analyse why those
tickets exist. Are there recurring themes? Are we making the same mistake
in multiple parts of the codebase? Is it possible that the same problem is
being made somewhere else, but not being reported?

Yours,
Russ Magee %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [GSoC 2013] Composite fields: first draft of the proposal

[Andrew Godwin]

Hi Michal,

This looks like a good starting point for a proposal (not to mention that
we don't doubt that you know about the problem area here!) - a few comments:

 - I'd like a bit more detail on some of your timeline points, in
particular what the introspection parts and primary key updates are
(personal bias, I suspect)
 - I'd like more details on what's been holding it back since GSOC 2011,
and how this GSOC is going to help serve those

Still, I like the general idea. You've mentioned tests and docs too. And I
really, really want to land this feature!

Andrew


On Fri, Apr 19, 2013 at 10:17 PM, Michal Petrucha wrote:

> Hello,
>
> as I promised a week ago, I'm posting a draft of my proposal. It is
> far from complete, specifically, it does not contain technical details
> on how things are going to be implemented. I'll try to fill these in
> as soon as possible, although I wrote more on most of them in my
> previous emails to this list. The most important thing is that it
> contains a timeline and a description of possible fallbacks.
>
> The proposal is accessible as a public gist [1] where it will be
> updated in the future, I'm posting the full text here as well for
> convenience.
>
> Have a nice weekend.
>
> Michal
>
> [1] https://gist.github.com/konk/5408673
>
>
> Composite Fields, vol. 2
> 
>
> Aim of this project
> ===
>
> Since I started working on this two years ago, I managed (with the
> help of a few people) to get most of the hard work done. The biggest
> part that I didn't get around to implementing was support in the ORM
> for multi-column joins. This has, however, been implemented recently
> by Jeremy Tiller and Anssi, which means there are only a few things
> left to be done.
>
> First of all, the code sitting in my github repo is badly out of date,
> which means it needs to be updated to the current state of master.
> While I'm at it, I also want to clean up the revision history to get
> it as close to a state where it could be just reviewed and merged
> directly as possible.
>
> Beside this, there are only the juicier features left that we
> initially wanted to leave unsupported and get back to them later.
> Those are the following (in no particular order):
>
> - ``GenericForeignKey`` support for ``CompositeField``
> - more intelligent handling of ``__in`` lookups
> - database instrospection and ``inspectdb``
> - detection of a change of the primary key in model instances
>
> Timeline
> 
>
> Our exam period starts on May 20. and ends on June 28., which means
> that I can't guarantee I will be able to fully dedicate myself to the
> project for the first two weeks, however, if nothing goes wrong, I
> should be able to pass all exams before June 17.
>
> Also, I intend to go to EuroPython, which means the first week of July
> won't be as fruitful as the others, but otherwise I'm ready to work
> full time on the project.
>
> Week  1 (Jun 17. - Jun 23.)
> ~~
> - porting the required virtual field changes, like a ``VirtualField``
>   class and ``django.db.models.options`` changes
> - revisiting the documentation I wrote two years ago to reflect the
>   evolution this project has gone through
>
> Week  2 (Jun 24. - Jun 30.)
> ~~
> - porting the virtual ``ForeignKey`` patchset on top of master to get
>   most of the functionality right
>
> Week  3 (Jul  1. - Jul  7.)
> ~~
> - EuroPython 2013, Florence
> - I intend to spend the full two days of sprints working on this but I
>   can't promise much more during this week.
> - running through the tests and fixing those that need updating,
>   ironing out the remaining wrinkles
>
> Week  4 (Jul  8. - Jul 14.)
> ~~
> - basic ``CompositeField`` functionality, ``CompositeValue`` and
>   descriptor protocol
>
> Week  5 (Jul 15. - Jul 21.)
> ~~
> - ``db_index``, ``unique`` and ``primary_key`` for ``CompositeField``
> - backend-dependent SQL for ``__in`` lookups on ``CompositeField``
>
> Week  6 (Jul 22. - Jul 28.)
> ~~
> - the few patches required to make admin work with composite primary
>   keys
>
> Week  7 (Jul 29. - Aug  4.)
> ~~
> - composite ``ForeignKey``: basic functionality, descriptors, database
>   JOINs
>
> Week  8 (Aug  5. - Aug 11.)
> ~~
> - composite ``ForeignKey``: customization of auxiliary fields,
>   adapting ``ModelForms`` and the admin in case it is necessary
>
> Week  9 (Aug 12. - Aug 18.)
> ~~
> - ``GenericForeignKey`` and ``GenericRelation`` support
>
> Week 10 (Aug 19. - Aug 25.)
> ~~
> - database introspection: create composite fields where necessary
>
> Week 11 (Aug 26. - Sep  1.)
> ~~
> - database introspection: composite ``ForeignKey``
>
> Week 12 (Sep  2. - Sep  8.)
> 

Re: [GSoC 2013] Improved error reporting

[Andrew Godwin]

Hi Damian,

We don't generally accept GSOC projects that are just a bit grab bag of
problems - this is looking a little bit like that. I'd like to see a better
breakdown of what kind of time each ticket would take and what your
planning schedule would be - in particular, I'd like to make sure you have
roughly 12 full weeks worth of work, as some of these tickets are quite
easy to fix.

It would also be nice to see you pick fewer issues but ones which were more
difficult - a past security GSOC just focused on three or four main,
difficult problems, and while not all got done, it developed in a
reasonable way.

Andrew


On Sat, Apr 20, 2013 at 11:00 PM, Damian Skrodzki wrote:

> *
>
> Hi again.
>
>
> Let’s try analyse the topic Improved error handling a little deeper.
>
>
> I’ve found dozen opened issues related to bad error handling (these are
> rather *not* from Better Error Messages list). In general most of the bugs
> demand different approach for each one.
>
> 1. django admin
>
> https://code.djangoproject.com/ticket/9373
>
> https://code.djangoproject.com/ticket/14642 - maybe patch from last
> comment will be enough here
>
> https://code.djangoproject.com/ticket/14408
>
> https://code.djangoproject.com/ticket/10919 - probably limiting number of
> displaying modified objects would fix the problem
>
> https://code.djangoproject.com/ticket/17417 -  it would worth to track
> and fix or raise suitable error
>
>
> 2. templates:
>
> https://code.djangoproject.com/ticket/18995
>
> 3. django.db
>
>
> https://code.djangoproject.com/wiki/BetterErrorMessages?version=48#django.db- 
> quite complex problem with high severity
>
> https://code.djangoproject.com/ticket/13776 - [related to above] this
> could be fixed after some core dev will make decision how this code should
> behave (if it’s worth to pass this task from someone that will understand
> the problem to someone else)
>
> 4. various:
>
> https://code.djangoproject.com/ticket/20250 - probably NoneType should be
> detected in django.db.models.sql.where.as_sql(self, qn, connection) already
> or even before. If we don’t want to prevent this value to be None or it’s
> not easily possible, the solution would be to catch None in last function
> raising some error. Error message could contain tip saying what might solve
> this problem.
>
> https://code.djangoproject.com/ticket/16840 - need to raise an error for
> unexisting fields
>
> https://code.djangoproject.com/ticket/19560 - consider if the warning
> should be changed into error here
>
> https://code.djangoproject.com/ticket/15063 - I don’t fully understand
> the comment  2 but possibly I could fix it
>
> 5. group of tickets “approved” by core developers to produce more verbose
> message :
>
> https://code.djangoproject.com/ticket/18959
>
> https://code.djangoproject.com/ticket/18866
>
> https://code.djangoproject.com/ticket/14343
>
> https://code.djangoproject.com/ticket/12756
>
> https://code.djangoproject.com/ticket/8760
>
>
> Plan what to do:
>
> Fixing most of errors should consist of following steps:
>
> 1. Analyse where and why error occurs.
>
> 2. Try to reproduce the bug (the best way would be creating a test if it’s
> not provided)
>
> 3. Now take one of actions:
>
>  a. Just fix error message
>
>  b. Consult issue with core developer and fix the problem raising error
> in more proper place or way
>
> 4. Create a fix for ticket and submit a patch to review
>
>
> Schedule
>
>
> I thought about splitting the time for few-days intervals (it might be
> weeks) and assigning that bugs to them. The most important problems should
> be resolved first. As I said: If i take an easier set of issues - it should
> be possible to fix a dozen of them. If I’ll take more complex tickets that
> demand to take some decisions and medium reworks - I should take just few.
> Taking few sounds more reasonable for me.
>
>
> Committing
>
>
> I’m going to focus on a bunch of small problems so here I propose to do a
> commit (pull request) per problem. This would be easier to review by other
> developers and easier to link with problem. Additionally if some of the
> fixes will need additional changes, it won’t have any influence on the
> others.
>
>
> Questions
>
> 1. Do you guys know some other “known” errors which are confusing to users?
>
> 2. Could this task include some things like:
>
> https://code.djangoproject.com/ticket/18171 - some steps have been taken
> but it is still unfixed (and related to raised exception)
>
> https://code.djangoproject.com/ticket/17015 - another issue that requires
> bigger patch and finally causes confusing error message.
>
> or I rather shouldn’t touch errors like this?
>
>
> What next?
>
>
> I think finally I should take few or dozen of problems. My proposals here
> are weekly covering with your exemplary areas except admin area. Am I still
> on a correct track? There are few tickets related to django admin, maybe
> focusing on them will be a good direction. Although