Makemigrations hooks for third-party apps?
Hi there I'm in the early stages of developing a third party app which makes heavy use of postgres extensions. The app provides an abstract model, so I won't have direct control of user's models or migrations. I'm having trouble getting it to generate sensible migrations for users. At present the migration process for users is going to be "copy and paste my migration file template into your project, and change the model names to match your app". As far as I can tell there is no way to generate migrations with anything custom. The operations I need to generate are: - CreateExtension() - RunSQL() - (to create slightly non-standard constraints) e.g. perhaps my model could supply a couple of methods: - Model.Meta.pre_creation_operations() - returns a list of operations to apply before the auto-detected model operations, e.g. creating extensions etc - Model.Meta.post_creation_operations() - the same but applied after the auto-detected operations, e.g. adding constraints/indexes. I'm envisaging that the operations returned from both methods would be deconstructed and copied into migration files when makemigrations is run. Has there been any discussion about this previously? I haven't found any. Thanks Craig de Stigter -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CANv5zmWUsfEWpqUWh6He0mMgnt3Pen-oKq%3Dw0wghPzwwtgWWyA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: Why not Single Table Inheritance?
Late reply I know but I see a lot of FUD in this thread and I want to try and clear it up. > in the general case, STI means you have to make almost all the fields in your model NULLable. You lose any semblance of having an actual database schema, and end up writing a whole lot of code to re-implement the features of a database schema instead of using the well tested, robust implementation that the database provides. This is just not true. Well implemented STI does *not* turn your database table into a key-value store. You can use the fields of the table just like you would normally use them. The *only* abnormal consideration here is that, if you *need* for some reason to have a field defined on a child class instead of all fields defined on the base model, then such fields must be nullable. In my experience users of django-typed-models will have 90-100% of their fields defined on the base class for each abstract type, meaning that only 0-10% of the fields will have to be nullable. There is no deviation from normal use of the ORM, and no hoops to jump through to ensure db consistency. In my own case, I am using django-typed-models with all the fields on the base class. In fact, django-typed-models didn't initially allow for fields on child models at all, and I still don't use that behaviour. I did merge a PR to add the possibility for fields defined on child models, and I can see how that would be useful. If you use that functionality you just have to be okay with those fields being nullable. > Django was designed around PostgreSQL, a database that cares about its data. And I wouldn't use MySQL if you paid me. But this is a straw man; if you're using STI you *might* be using fields that are nullable but could otherwise be non-nullable, but this shouldn't imply that you don't care about your data. I reserve judgment on whether STI should be included in core. It works fine in a third-party app. Some better support for it in core would be helpful, since currently I'm relying on some unsupported stuff that the core devs could decide to break at any time. But it's a better solution to certain problems than MTI is, and it doesn't deserve the bashing that some people seem to give it. Use the right tool for the job. If that's STI then use STI :) Craig de Stigter On Saturday, 7 June 2014 06:05:29 UTC+12, Aymeric Augustin wrote: > > On 6 juin 2014, at 09:42, Thomas Güttler > > wrote: > > > I think it is a "not invented here" syndrome: Ruby on Rails did it > before. That's > > a reason to do it different. > > The reason is more simple. > > Rails was designed around MySQL, a database with a rather casual > relationship > to data integrity. It will happily truncate data or save invalid values in > the name of > performance. In the same spirit STI trades data integrity for speed. It > avoids joins, > which can be very slow on MySQL, but also prevents the database from > enforcing > constraints. > > Django was designed around PostgreSQL, a database that cares about its > data. > > That explains many differences in the design of the Rails and Django ORMs. > > -- > Aymeric. > > > > > -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/d5114d13-a1d2-4f28-85c6-1771d3896f2e%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Why not Single Table Inheritance?
> If you ignore STI, I think it is quite straightforward to solve this with a parent model class which adds a type field, and manager methods to add the select_related calls and "interpret" the type field properly; so I don't see an immediate need for inclusion in core. Well, you don't need select_related calls at all, if you're actually storing things in one table like "single-table inheritance" implies. I too was surprised to find Django doesn't do this, and was unable to find a good third-party app that does it. So I wrote my own: https://github.com/craigds/django-typed-models/ It works well and we have been using it in production for a couple years. It does rely on a few hacks that Django doesn't officially support, like proxy models with their own fields, which has unfortunately been broken in django 1.7 <https://code.djangoproject.com/ticket/22690>. I'd love to see better support for this in Django core. Regards Craig de Stigter On Thursday, 22 May 2014 21:02:48 UTC+12, Anssi Kääriäinen wrote: > > On 05/22/2014 11:13 AM, Shai Berger wrote: > >> Any thoughts on this idea? > >> > > Instinctively -- isn't it possible to achieve the same things today by > > overriding __new__ ? > My understanding is that achieving all the same things isn't possible. > The problem is that inside __new__ it is impossible to know if the call > to __new__ was made from database loading or from user code. It also > seems that it is impossible to alter the args and kwargs passed to > __init__(). In addition if one wants for some reason (speed, or not > invoking __setattr__) to assign values directly to the __dict__ of the > new class, then __new__() doesn't seem to offer any way to do that. > > It is true that STI is likely possible with usage of __new__() as long > as you don't want to change the arguments to the __init__ call of the > created object. > > As a side note I think direct assignment to __dict__ on model loading > would be a better design than the current __init__ call. For example > Django needs to do some pretty crazy stuff in __init__() to support > deferred field loading. With direct __dict__ assignment deferred model > creation is trivial. Also, loading from the database is a form of > deserialization, and when deserializing you want to load the model as it > were saved. The way to do this is to avoid __init__, __setattr__ and > descriptor __set__ calls. To avoid those the values should be assigned > directly to the __dict__ of the object. This is also used by Python's > deserialization. Of course, thinking about this is mostly academic. > Changing the way model loading from database is done has severe > backwards compatibility issues. Even django-core relies on descriptor > calls in some case. As an example to_python() method of custom fields is > called through a descriptor. > > - Anssi > -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/ffb11c40-c231-48d5-898a-71684536f55e%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: memory leak in django 1.5
Just confirming that this fixed the memory leak problem for us. Thanks again :) On Friday, March 22, 2013 3:07:02 PM UTC+13, Craig de Stigter wrote: > > Karen Tracey saves the day! > > Thanks so much, seems likely that's it :) > > Craig de Stigter > > On Friday, March 22, 2013 2:25:04 PM UTC+13, Karen Tracey wrote: >> >> On Thu, Mar 21, 2013 at 9:00 PM, Craig de Stigter wrote: >> >>> Hi everyone >>> >>> (cross-posted because this seems relevant to both django-users and >>> developers, and both might have experienced this problem) >>> >>> We've noticed a gradual increase in memory usage for our apache >>> processes since upgrading to django 1.5. >>> >>> Here is a graph <http://i.imgur.com/cBsUx52.png> showing memory usage >>> before and after we upgraded to django 1.5 on our web server. (the upgrade >>> occurs at 12:30 on the graph). >>> >>> I'm hoping someone else has noticed a similar thing and is able to >>> provide some insight. >>> >> >> https://code.djangoproject.com/ticket/19895#comment:6 >> >> notes a memory leak due to a fix that went into 1.5. >> >> The fix was reverted on the 1.5.x branch about two days ago, so one thing >> to try would be to run current 1.5.x branch level rather than released 1.5. >> >> Karen >> > -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: memory leak in django 1.5
Karen Tracey saves the day! Thanks so much, seems likely that's it :) Craig de Stigter On Friday, March 22, 2013 2:25:04 PM UTC+13, Karen Tracey wrote: > > On Thu, Mar 21, 2013 at 9:00 PM, Craig de Stigter > > > wrote: > >> Hi everyone >> >> (cross-posted because this seems relevant to both django-users and >> developers, and both might have experienced this problem) >> >> We've noticed a gradual increase in memory usage for our apache processes >> since upgrading to django 1.5. >> >> Here is a graph <http://i.imgur.com/cBsUx52.png> showing memory usage >> before and after we upgraded to django 1.5 on our web server. (the upgrade >> occurs at 12:30 on the graph). >> >> I'm hoping someone else has noticed a similar thing and is able to >> provide some insight. >> > > https://code.djangoproject.com/ticket/19895#comment:6 > > notes a memory leak due to a fix that went into 1.5. > > The fix was reverted on the 1.5.x branch about two days ago, so one thing > to try would be to run current 1.5.x branch level rather than released 1.5. > > Karen > -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: memory leak in django 1.5
Karen Tracey saves the day! Thanks so much, seems likely that's it :) Craig de Stigter On Friday, March 22, 2013 2:25:04 PM UTC+13, Karen Tracey wrote: > > On Thu, Mar 21, 2013 at 9:00 PM, Craig de Stigter > > > wrote: > >> Hi everyone >> >> (cross-posted because this seems relevant to both django-users and >> developers, and both might have experienced this problem) >> >> We've noticed a gradual increase in memory usage for our apache processes >> since upgrading to django 1.5. >> >> Here is a graph <http://i.imgur.com/cBsUx52.png> showing memory usage >> before and after we upgraded to django 1.5 on our web server. (the upgrade >> occurs at 12:30 on the graph). >> >> I'm hoping someone else has noticed a similar thing and is able to >> provide some insight. >> > > https://code.djangoproject.com/ticket/19895#comment:6 > > notes a memory leak due to a fix that went into 1.5. > > The fix was reverted on the 1.5.x branch about two days ago, so one thing > to try would be to run current 1.5.x branch level rather than released 1.5. > > Karen > -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
memory leak in django 1.5
Hi everyone (cross-posted because this seems relevant to both django-users and developers, and both might have experienced this problem) We've noticed a gradual increase in memory usage for our apache processes since upgrading to django 1.5. Here is a graph <http://i.imgur.com/cBsUx52.png> showing memory usage before and after we upgraded to django 1.5 on our web server. (the upgrade occurs at 12:30 on the graph). I'm hoping someone else has noticed a similar thing and is able to provide some insight. I've spent two days trying to track down where the leak might be but so far come up empty. We are using modwsgi on ubuntu lucid. We were previously running a pre-release (November) django 1.5a1 with no problems, so the bug must be somewhere in https://github.com/koordinates/django/compare/31e616521...e4ba16ce7 . Unfortunately that's a big diff. I've been unable to reproduce the bug on a dev server, and I'm not comfortable with doing `git bisect` on our production box ;) I've tried using Dozer to track down the leak but it appears to cause apache to segfault with annoying regularity before finding anything of consequence. Any tips welcome Thanks Craig de Stigter -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: Single table inheritance - working implementation
Hi Krzysztof and everyone else Thanks! Your fork looks very promising. I'll review the fork today and possibly pull changes in if the changes aren't too disruptive. Some more tests would probably be good. I've added a New BSD License<https://github.com/craigds/django-typed-models/blob/master/LICENSE.txt> to the repo to clear up any confusion. Would other devs be interested in adding something like django-typed-models to Django core? What kinds of changes would make that more likely? Cheers Craig de Stigter On Tue, Oct 30, 2012 at 2:34 AM, Krzysztof Jurewicz < krzysztof.jurew...@gmail.com> wrote: > Hi all, > > In February, Craig de Stigter released > django-typed-models<https://groups.google.com/d/topic/django-users/63z-ejUQ1Eg/discussion>, > a package implementing inheritance hierarchy for Proxy models with > automatic type recasting. I've written a fork of it with additional > possibility of adding fields to subclasses, therefore achieving a single > table inheritance implementation similar to Ruby on Rails' STI. This is > possible mostly thanks to metaclass magic and monkey patching. > > The fork's homepage is at https://github.com/KrzysiekJ/django-typed-models. > Note that it should be considered early alpha and there is much room for > optimisation, code cleanup and maybe additional features. Feel free to fork > and/or add issues. > > I don't know yet what to do with licensing since Craig didn't specify any > license in the original package. Craig, please comment or send me an email. > > Regards, > Krzysztof Jurewicz > -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Incorrect serialization for 3d GEOSGeometry
Hi folks I discovered django.contrib.gis support for 3d geometries is pretty patchy. I realised that the database stuff only worked in postgis, etc, but it turns out even the serialisation methods on GEOSGeometry don't work properly. Here's a small test script that demonstrates the problem: https://gist.github.com/2594905 And here's the output: https://gist.github.com/2594926 In summary, all these properties return 2d-only output for 3d geometries (the Z dimension gets stripped): - wkt - ewkt - hex - wkb - json - geojson Yet these others produce correct 3d output: - ewkb - hexewkb - kml Is there any reasoning behind this, or should I create a ticket and try to fix them? I tried posting this to the geodjango list (here)<https://groups.google.com/forum/#!msg/geodjango/XQkkeDmsCDI/ZBYq1XZ6SngJ>, but I'm not sure if anyone actually reads those posts, as I haven't got a response in 10 days. Thanks Craig de Stigter -- You received this message because you are subscribed to the Google Groups "Django developers" group. To view this discussion on the web visit https://groups.google.com/d/msg/django-developers/-/s4smWq7j4xIJ. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: type-field model inheritance
Since multi-table-inheritance is the only kind of inheritance (apart from abstract/proxy) supported by Django's ORM, I don't know what other type of inheritance django_polymorphic would be referring to... As per my original post, I want to store everything in one table (all subclasses have the same fields). I just want to have different python behaviour for subclasses. Basically proxy models but with a known type and polymorphic querysets. I still haven't looked into django_polymorphic to see if it does that, but it seems like it could easily... Hopefully I'll get time to play with it in the next couple days. Regards Craig -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: type-field model inheritance
Hi guys Thanks for pointing those out. I knew I couldn't have been the first to want this. I guess I just didn't know the right words to search for here. It looks like django_polymorphic does what I want. I'm not yet sure why it says it takes one query per type of model in a queryset. Unless it is talking about multi-table inheritance, in which each type would require a different join. But I'll look in more detail in the next few days and no doubt it will become clear. Thanks for the reference to CORBA narrow too. Skimming the thread now. Regards Craig -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: type-field model inheritance
I realise everyone's been busy with getting 1.3 ready, but doesn't anyone
have thoughts on this? It's been two weeks ...
Thanks
Craig
On Thursday, February 17, 2011 11:08:57 PM UTC+13, Craig de Stigter wrote:
>
> Hi folks
>
> Ever since Django started supporting various types of model inheritance
> I've wondered why it lacks the kind that I would find most useful: python
> behaviour differentiated based on the value of a field.
>
> I'll explain with an example. Here's what I'd like to do:
>
> class Datasource(models.Model):
> type = models.ModelTypeField()
> uri = models.CharField(max_length=256)
>
> # common behavior in the superclass
> def __repr__(self):
> return u'<%s: %s>' % (self.__class__.__name__, self.uri)
>
> class HttpDatasource(Datasource):
> # custom behaviour in the subclasses
> def get_filename(self):
> return self.uri.rsplit("/", 1)[-1]
>
> class ZipfileDatasource(Datasource):
> def get_filename(self):
> files = zipfile.list(self.uri)
> return files[0].rsplit('/', 1)[-1]
>
> >>> zip = ZipfileDatasource.objects.create(uri="/path/to/foo.zip")
> >>> uri = UriDatasource.objects.create(uri="http://example.com/foo.txt";)
>
> >>> Datasource.objects.all()
> [, http://example.com/foo.txt>]
>
> >>>ZipfileDatasource.objects.all()
> []
>
> >>> Datasource.objects.all().values_list('type', flat=True)
> [u'myapp.models.ZipfileDatasource', u'myapp.models.UriDatasource']
>
>
> These are quite similar to proxy models, but vary in their queryset
> behaviour - the generic Datasource queryset has mixed types and the concrete
> querysets are always filtered by type.
>
> This is far more useful than proxy models, since the concrete types of each
> table are known. It's also better than making an abstract model and
> subclassing it, because now all the objects are in the same table and you
> can iterate over them all at once if you want. Adding more types is easy,
> since there are no schema changes (with abstract models you'd have to add a
> new table for each type).
>
> It's possible that proxy models could be extended to do this without
> breaking existing code. I'm not sure yet.
>
> I'm thinking of diving into this. Does anyone have any suggestions? Or,
> perhaps there's a reason why this hasn't been done previously that you more
> experienced gurus could illuminate?
>
> Thanks in advance
>
> Craig de Stigter
> Maintainer of django-mptt and full time dev at koordinates.com
>
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
type-field model inheritance
Hi folks
Ever since Django started supporting various types of model inheritance I've
wondered why it lacks the kind that I would find most useful: python
behaviour differentiated based on the value of a field.
I'll explain with an example. Here's what I'd like to do:
class Datasource(models.Model):
type = models.ModelTypeField()
uri = models.CharField(max_length=256)
# common behavior in the superclass
def __repr__(self):
return u'<%s: %s>' % (self.__class__.__name__, self.uri)
class HttpDatasource(Datasource):
# custom behaviour in the subclasses
def get_filename(self):
return self.uri.rsplit("/", 1)[-1]
class ZipfileDatasource(Datasource):
def get_filename(self):
files = zipfile.list(self.uri)
return files[0].rsplit('/', 1)[-1]
>>> zip = ZipfileDatasource.objects.create(uri="/path/to/foo.zip")
>>> uri = UriDatasource.objects.create(uri="http://example.com/foo.txt";)
>>> Datasource.objects.all()
[, http://example.com/foo.txt>]
>>>ZipfileDatasource.objects.all()
[]
>>> Datasource.objects.all().values_list('type', flat=True)
[u'myapp.models.ZipfileDatasource', u'myapp.models.UriDatasource']
These are quite similar to proxy models, but vary in their queryset
behaviour - the generic Datasource queryset has mixed types and the concrete
querysets are always filtered by type.
This is far more useful than proxy models, since the concrete types of each
table are known. It's also better than making an abstract model and
subclassing it, because now all the objects are in the same table and you
can iterate over them all at once if you want. Adding more types is easy,
since there are no schema changes (with abstract models you'd have to add a
new table for each type).
It's possible that proxy models could be extended to do this without
breaking existing code. I'm not sure yet.
I'm thinking of diving into this. Does anyone have any suggestions? Or,
perhaps there's a reason why this hasn't been done previously that you more
experienced gurus could illuminate?
Thanks in advance
Craig de Stigter
Maintainer of django-mptt and full time dev at koordinates.com
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
