Re: Composite primary keys
On Wed, 16 Mar 2011 07:49:17 +0200, Michal Petrucha wrote: On Tue, Mar 15, 2011 at 09:15:29PM -0500, Javier Guerra Giraldez wrote: On Tue, Mar 15, 2011 at 7:14 PM, Christophe Pettus wrote: > A concern here is that composite indexes, like unique, are > sensitive to the ordering of the fields, which means that the > ordering of the fields in the class declaration becomes important. This is the same reason a new Meta flag has been agreed upon in the past. (That, however, does not mean it has to be that way.) a simplistic proposal: the order of the fields on a composite index is determined by the exact value given to the primary_key argument. that way, just setting primary_key=True on a few fields won't guarantee order, but something like: class City (models.Model): country = models.CharField (max_length=2, primary_key=1) state = models.CharField (max_length=2, primary_key=2) city = models.CharField (max_length=3, primary_key=2) Name = models.CharField (max_length=20) would set the (country,state,city) primary key in the obvious order. in short: giving any non-falsy value to primary_key would add the field to the key, the exact value would determine ordering. I like this proposal. It might even be easier to implement than fiddling with new Meta flags and everything. One minor detail though, just setting primary_key=True on multiple fields would still have to guarantee some ordering since the primary key would be represented by a tuple. If you don't know for sure in which order the values are, you can't really use the .pk property. This feels like something that wants a named-tuple (or a full blown dict). Alternatively, provide a method on the model class that takes the name=value arguments (queryset style) and returns the right pk tuple. Otherwise, the exact ordering of the fields in the pk tuple becomes yet another implicit(!) part of the model's contract - and any code that wants to use this model will be that much more brittle. This would require a thorough explanation in the docs, that if you supply values that compare equal to primary_key, the order of fields will be the same as in the model definition. Michal Petrucha -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Composite primary keys
On Tue, Mar 15, 2011 at 09:15:29PM -0500, Javier Guerra Giraldez wrote: > On Tue, Mar 15, 2011 at 7:14 PM, Christophe Pettus wrote: > > A concern here is that composite indexes, like unique, are > > sensitive to the ordering of the fields, which means that the > > ordering of the fields in the class declaration becomes important. This is the same reason a new Meta flag has been agreed upon in the past. (That, however, does not mean it has to be that way.) > a simplistic proposal: > > the order of the fields on a composite index is determined by the > exact value given to the primary_key argument. > > that way, just setting primary_key=True on a few fields won't > guarantee order, but something like: > > class City (models.Model): > country = models.CharField (max_length=2, primary_key=1) > state = models.CharField (max_length=2, primary_key=2) > city = models.CharField (max_length=3, primary_key=2) > Name = models.CharField (max_length=20) > > would set the (country,state,city) primary key in the obvious order. > > in short: giving any non-falsy value to primary_key would add the > field to the key, the exact value would determine ordering. I like this proposal. It might even be easier to implement than fiddling with new Meta flags and everything. One minor detail though, just setting primary_key=True on multiple fields would still have to guarantee some ordering since the primary key would be represented by a tuple. If you don't know for sure in which order the values are, you can't really use the .pk property. This would require a thorough explanation in the docs, that if you supply values that compare equal to primary_key, the order of fields will be the same as in the model definition. Michal Petrucha signature.asc Description: Digital signature
Re: Expensive queryset cloning
In our profiling we've also noticed the cloning to be one of the slowest parts of the app (that and instantiation of model objects). We haven't yet, but had been planning on exploring a way to mutate the existing class in most circumstances, but haven't dug into it too much yet. On Mar 14, 11:16 pm, Alexander Schepanovski wrote: > > I'd be surprised if the cloning of querysets is actually a > > significant bottleneck relative to the database query itself > > I was too. > Query via ORM is 2-4 times slower for me than just database query + > network roundtrip. It is not only due queryset cloning, but cloning > takes 0.5-1 of that 2-4 times. > Also, I cache db queries issued via ORM, so for cache hit I need to > construct queryset but don't need to execute query. > > > unless you're doing loops with hundreds or thousands of clones, in which > > case > > it can almost certainly be optimized via Q objects. > > not loops but many ifs. I already use dicts and Q objects (when need > complex conditions) but I still have no less than 3 (filter + order_by > + slice) clones for each query. Here we see an interesting thing - > cloning querysets made me write less beautiful code. > > > If you really think it will make a significant difference for your app, > > you can probably achieve this for yourself by using your own subclass of > > QuerySet and overriding the _clone method. > > Already do this, but it does not cover User model and such. I probably > go with some monkey patch for now. > > > This is not an option. It will break quite a lot of existing code, and > > often in highly confusing ways. > > Not so much code. In most cases cloning is not needed. But it will be > confusing breakage, I agree. On other hand for an unbiased person > cloning can be confusing too. I wonder how many people were surprised > why > qs.filter(...) > does not filter qs. > > Ok, if not by default, still we can provide some .mutating() method > or .cloning(bool) method which switches queryset into/off mutating > mode. Then we can use it in some django code such as Manager.get to > avoid useless cloning. -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Wrong error message when user having is_staff=False tries to login to admin
I also don't think it should be considered a security vulnerability to reveal that an authenticated user does not have permission to access the admin (or any other) app. If the credentials are valid and they authenticate against the defined authentication backends, then we should assume that we are talking to a trusted authenticated user and we should present error messages that are at the very least not misleading. Assuming that any authenticated user might be an attacker who has brute forced a password and presenting obscure error messages to authenticated users is not helping anybody. Cheers. Tai. On Mar 16, 3:41 am, "Brian O'Connor" wrote: > 2011/3/15 Juan Pablo Martínez > > > The admin is not "one more app" is (if I may) the app with more weight > > on most sites. Someone who has access to the admin has access to most > > or all information. There is no "one more app. " > > This has nothing to do with the argument here. The account in question, as > already stated many times, has no access to the admin site. That's the > whole point of this discussion. > > Carelessness or neglect of a click in the admin should't call into> question > the admin with the justification "that does not happen > > again." > > This has to do with deliberately misleading users. I've been stuck by this > at least once in my django career, and artemy has too. People make > mistakes, it happens. > > > I think if everyone is going to fix "contrib" to your needs the > > contrib lost all independence. > > I especially don't understand this statement. The whole point of > django-developers is to discuss development of django, and by extension > (because there are no other lists, as far as I'm aware) the contrib > modules. Everyone comes here to help make the project better, to help fit > their needs. That's the whole point, as far as I'm concerned. > > A reasonable suggestion was made, in which a few people came back and said > that by doing this improvement, it would open a security issue. Myself, and > others have stated that in fact, this would not be a security issue, and > have provided examples. > > At this point, I'll absolutely never forget to check the is_staff flag > purely because I've been following this discussion. What I don't understand > is why there is such a huge opposition to the change. > > -- > Brian O'Connor -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Composite primary keys
On Tue, Mar 15, 2011 at 7:14 PM, Christophe Pettus wrote: > A concern here is that composite indexes, like unique, are sensitive to the > ordering of the fields, which means that the ordering of the fields in the > class declaration becomes important. a simplistic proposal: the order of the fields on a composite index is determined by the exact value given to the primary_key argument. that way, just setting primary_key=True on a few fields won't guarantee order, but something like: class City (models.Model): country = models.CharField (max_length=2, primary_key=1) state = models.CharField (max_length=2, primary_key=2) city = models.CharField (max_length=3, primary_key=2) Name = models.CharField (max_length=20) would set the (country,state,city) primary key in the obvious order. in short: giving any non-falsy value to primary_key would add the field to the key, the exact value would determine ordering. -- Javier -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Composite primary keys
On Mar 15, 2011, at 5:06 PM, Russell Keith-Magee wrote: > And if you mark > multiple fields, then you have a composite primary key composed of > those fields. A concern here is that composite indexes, like unique, are sensitive to the ordering of the fields, which means that the ordering of the fields in the class declaration becomes important. That could, potentially, be surprising. -- -- Christophe Pettus x...@thebuild.com -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Composite primary keys
On Tue, Mar 15, 2011 at 11:11 PM, Andrew Godwin wrote: > On 14/03/11 21:14, Michal Petrucha wrote: >> >> Good evening (or whatever it is in everyone's timezone). >> >> I'm an undergrad computer science student at the Faculty of >> Mathematics, Physics and Informatics, Commenius University, >> Bratislava, Slovakia and I'm willing to participate in this year's >> GSoc. I'm interested in fixing the six-year-old open ticket in trac >> concerning the subject, http://code.djangoproject.com/ticket/373 > > Firstly, thanks for proposing a GSoC project - it's always nice to have more > students. Bear in mind that the list of accepted organisations hasn't been > published yet, but we're pretty hopeful Django will make it in again this > year. I'd like to second what Andrew has said. This is definitely a project with enough meat to fill a GSoC, and it's something that has been on Django's wish list for a long time. I also agree with Andrew's answers to your questions, with one exception: >> - The composite primary key would be specified as a tuple of strings >> in a primary_key attribute inside a model's Meta class instead of >> having a field with primary_key=True. > > That seems to match with our current handling of things like unique, so that > seems fairly reasonable. The difference between primary_key and unique is that you can have multiple unique conditions, including a number of individual field uniques *plus* a series of grouped uniques. However you can only have one primary key field (or field set). I'd like to suggest an alternate representation. At present, you can make any field a primary key by setting primary_key=True on that field. The Meta class then validates to ensure that there is only one field that has that attribute set (and inserts an AutoField if no field has it set). Another way to represent primary keys would be to relax this validation constraint. If you mark no field as primary_key=True, then an AutoField is added. If you mark one field, then that field is the primary key. And if you mark multiple fields, then you have a composite primary key composed of those fields. This removes the need to introduce a new Meta flag, but more importantly, it means you won't have to resolve discrepancies between fields with primary_key and a Meta.primary_key option. Yours, Russ Magee %-) -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Homogenization of User and UserProfile
Hello developers, I've been using Django for a couple of months now on a new project and I've come across something which I think might be improved upon. It deals with the nature of UserProfile, which I think is a very handy tool overall. The thing I'm not totally satisfied with is the way that UserProfile and User are totally separate objects once you have them at the python level. An example of an area where this separation might be an issue is making a JSON web service that needs to send a full User object to a client. In a typical client app, your User object is most likely one robust object which you have to express in Django as both User and UserProfile. So when you go to pipe this information through a tool like Django-Piston, you have to do a bit of slicing and dicing in your handler code to get both objects through in the same request, or manually create a nice neat object for Piston to deliver to the client in a way the client will be able to consume without having to worry about matchmaking Users and Profile.user_id fields. Now, I am really not even advanced in Python, and I've only been using Django for a couple of months, but it seems to me like having a convenience method similar to get_profile() that automatically enumerates through your profile object and produces a flattened version of your client friendly user object might be a very useful thing to have. Maybe there is already an easy and kosher way to do this in Python that I am unaware of, regardless, it just seems like something that would be handy to have for people who deal primarily with using Django to serve non-web clients. I don't have a good sense of if this is something that is appropriate for Django Auth or should just be a property of my custom profile object, but I thought I might throw it out there for consideration. Thanks for your time, Michael -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: QuerySet subclass based on namedtuple()
Hi Alexander, On 03/15/2011 03:30 AM, Alexander Schepanovski wrote: > It would be nice to have a QuerySet subclass based on namedtuple(). > namedtuples takes less memory than dicts (from ValuesQuerySet), much > more convenient and induce more readable code than tuples > (ValuesListQuerySet). > > Namedtuples use same dot notation as model instances. So the same > simple enough code can operate on both. You can easily upgrade/ > downgrade your queryset when needed. This is an interesting idea. I think it falls into the category of things that can be implemented outside core as a reusable library to prove its usefulness before being considered as a core change. Carl -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Wrong error message when user having is_staff=False tries to login to admin
2011/3/15 Juan Pablo Martínez > The admin is not "one more app" is (if I may) the app with more weight > on most sites. Someone who has access to the admin has access to most > or all information. There is no "one more app. " > > This has nothing to do with the argument here. The account in question, as already stated many times, has no access to the admin site. That's the whole point of this discussion. Carelessness or neglect of a click in the admin should't call into > question the admin with the justification "that does not happen > again." > > This has to do with deliberately misleading users. I've been stuck by this at least once in my django career, and artemy has too. People make mistakes, it happens. > I think if everyone is going to fix "contrib" to your needs the > contrib lost all independence. > > I especially don't understand this statement. The whole point of django-developers is to discuss development of django, and by extension (because there are no other lists, as far as I'm aware) the contrib modules. Everyone comes here to help make the project better, to help fit their needs. That's the whole point, as far as I'm concerned. A reasonable suggestion was made, in which a few people came back and said that by doing this improvement, it would open a security issue. Myself, and others have stated that in fact, this would not be a security issue, and have provided examples. At this point, I'll absolutely never forget to check the is_staff flag purely because I've been following this discussion. What I don't understand is why there is such a huge opposition to the change. -- Brian O'Connor -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Wrong error message when user having is_staff=False tries to login to admin
The admin is not "one more app" is (if I may) the app with more weight on most sites. Someone who has access to the admin has access to most or all information. There is no "one more app. " Carelessness or neglect of a click in the admin should't call into question the admin with the justification "that does not happen again." If the application is used only in Intranet or not is entirely separate from whether to put the right message or not. I think if everyone is going to fix "contrib" to your needs the contrib lost all independence. 1.3 will create a special form and then you can change the message. I do not think the amendment is necessary. Regards, On 14/03/2011, Nick Phillips wrote: > On Mon, 2011-03-14 at 15:57 +, Tom Evans wrote: > >> This is one of my bug-bears with the current authentication system - >> it has no concept of role. The current action when an identified user >> visits the admin site is to display a login form, which is totally >> wrong in my opinion. The user has already presented credentials, which >> we have accepted, so why should we expect them to have another >> different set of credentials? >> >> Similarly, most restricted access views are currently protected by the >> login_required decorator. Again, this is pretty good, but it doesn't >> solve the authorization issue. With the vast majority of views I >> decorate with @login_required, I actually need three states: >> >> Unidentified -> login page >> Identified, but no access -> homepage, with error message >> Identified, access -> allow through > > Spot on, IMO - it's muddling authentication and authorization up > together. The user is authenticated, and shouldn't be arbitrarily given > the impression that they're not when they try to access a page to which > they don't have access. There's nothing "special" about the admin here, > except that it is provided as part of the "batteries included". How does > your site respond when a user tried to access any other URL to which > they are denied access? Whatever happens there is probably what should > happen when they try to access the admin. It could be a 403, a redirect > to homepage, a forced logout with disabling of account and email to > admin, could be all sorts of things, depending on the site. > > The question should probably be "how can we allow the developer to > specify the desired behaviour when the user attempts to access a URL to > which they have no access?" - subject to the usual provision of sensible > defaults etc. > > > Cheers, > > > Nick > -- > Nick Phillips / +64 3 479 4195 / nick.phill...@otago.ac.nz > # these statements are my own, not those of the University of Otago > > -- > You received this message because you are subscribed to the Google Groups > "Django developers" group. > To post to this group, send email to django-developers@googlegroups.com. > To unsubscribe from this group, send email to > django-developers+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/django-developers?hl=en. > > -- :: juanpex -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Composite primary keys
On 14/03/11 21:14, Michal Petrucha wrote: Good evening (or whatever it is in everyone's timezone). I'm an undergrad computer science student at the Faculty of Mathematics, Physics and Informatics, Commenius University, Bratislava, Slovakia and I'm willing to participate in this year's GSoc. I'm interested in fixing the six-year-old open ticket in trac concerning the subject, http://code.djangoproject.com/ticket/373 Firstly, thanks for proposing a GSoC project - it's always nice to have more students. Bear in mind that the list of accepted organisations hasn't been published yet, but we're pretty hopeful Django will make it in again this year. Before I dig deeper into the issue I would like to know whether there is interest in having this fixed and whether it is worth a full GSoC project. Also, is there already any work regarding this issue underway? If so, would it be reasonable for me to go on with this project? Anyway, if I'm to take on the task, there are quite a few design considerations to be taken care of. For starters, I've read through David Cramer's work on this from two-three years ago. I'd stick to the API skeleton that was agreed on back then, however, there remain lots of other unresolved questions. The following list is in no way meant to be exhaustive, it's just a list of things that came to my mind during the past hour. In fact, I'd appreciate other issues that would need to be kept in mind I forgot to mention. - The composite primary key would be specified as a tuple of strings in a primary_key attribute inside a model's Meta class instead of having a field with primary_key=True. That seems to match with our current handling of things like unique, so that seems fairly reasonable. - The pk property of model instances would be a tuple instead of a single value for composite key models. So the property will vary type based on the model? Obviously this is needed for backwards-compatability, but it's still going to affect e.g. library authors, so we'd need to make sure everything that previously accepted a PK also accepts a tuple. - The admin could reference composite keys using some kind of smart string escaping, for example escaping the , (comma) and using it as a delimiter. That should work, though it might be a bit ugly; again, I can't think of a better alternative. - This could maybe even be used in generic relations. Is it reasonable to support this in generic relations? The following post suggests it might not be the best idea: http://groups.google.com/group/django-developers/msg/dea0e360c6cd37a6 Generic relations are ugly and inefficient as it is; I'm not sure that including them in something like this is strictly necessary, much as Malcolm suggests, and probably outside the scope of a GSoC project (something of this scale is already touching a lot of places in the model layer as it is) - The managers and querysets would have to be updated to handle composite primary keys correctly. - Consequently, there would need to be added support in the SQL compiler. - The same holds for syncdb, inspectdb would be also nice. I'd say inspectdb isn't strictly necessary - not very many people use it. I'll also be looking at potentally merging in some schema alteration code (i.e. moving parts of the South database backends) over the next few months, so it's going to affect that, too, though I'm more than happy to help fix that part (the existing creation code will definitely need changing, at least). - ForeignKeys would have to be backed by multiple database columns. How should they be named by default? How should their names be overriden? Should db_column expect a tuple of strings? Should there be another db_column_prefix option to prefix the names with a common string? These are some very good questions, and something we need a good discussion here about; I'd personally say doing "fkname_remotecolname" is best for the columns (which then means "author_id" still matches most DBs, and you could also have something like "passport_country_id, passport_number"). I'm also -0 on the idea of a db_column_prefix, but db_column is going to need to take tuples/sequences for multi-column FKs. - What about the ForeignKey field's attname? Should it be a tuple of names or should it be a single string pointing to a tuple of attributes? I'd say a single string pointing to a tuple - I don't like fields which 'magically' make more attributes than the name they were assigned. That also makes sense if you think of the foreign key as a single entity composed of several values. - How should a person trying to add a ForeignKey field pointing to its own model into the primary_key be punished? Preferably with a model validation error. - Does it make sense to make a subset of columns created by a ForeignKey part of a primary key? Provided they're then mix
QuerySet subclass based on namedtuple()
It would be nice to have a QuerySet subclass based on namedtuple(). namedtuples takes less memory than dicts (from ValuesQuerySet), much more convenient and induce more readable code than tuples (ValuesListQuerySet). Namedtuples use same dot notation as model instances. So the same simple enough code can operate on both. You can easily upgrade/ downgrade your queryset when needed. I know namedtuples come only in python 2.6. We can provide a fallback or just don't support this for older pythons. -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Expensive queryset cloning
> I'd be surprised if the cloning of querysets is actually a > significant bottleneck relative to the database query itself I was too. Query via ORM is 2-4 times slower for me than just database query + network roundtrip. It is not only due queryset cloning, but cloning takes 0.5-1 of that 2-4 times. Also, I cache db queries issued via ORM, so for cache hit I need to construct queryset but don't need to execute query. > unless you're doing loops with hundreds or thousands of clones, in which case > it can almost certainly be optimized via Q objects. not loops but many ifs. I already use dicts and Q objects (when need complex conditions) but I still have no less than 3 (filter + order_by + slice) clones for each query. Here we see an interesting thing - cloning querysets made me write less beautiful code. > If you really think it will make a significant difference for your app, > you can probably achieve this for yourself by using your own subclass of > QuerySet and overriding the _clone method. Already do this, but it does not cover User model and such. I probably go with some monkey patch for now. > This is not an option. It will break quite a lot of existing code, and > often in highly confusing ways. Not so much code. In most cases cloning is not needed. But it will be confusing breakage, I agree. On other hand for an unbiased person cloning can be confusing too. I wonder how many people were surprised why qs.filter(...) does not filter qs. Ok, if not by default, still we can provide some .mutating() method or .cloning(bool) method which switches queryset into/off mutating mode. Then we can use it in some django code such as Manager.get to avoid useless cloning. -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
