Re: Using Cache+db for session backend
Hey Jessie -- See ticket #6791. It's got a well-written, working implementation; it didn't make it into 1.0 because of time constraints. It'll almost certainly make it into 1.1. Jacob --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Using Cache+db for session backend
I think it would be useful to include a session backend with Django that stores session data in the cache and falls back to the database if the session isn't in the cache. It's trivial to write your own (mine is about 30 lines of code), but it seems like such a common use case that it should just be in Django by default. Currently there are three session backends: cache, file, and db. They each have problems: * The cache-based session store that ships with django (django.contrib.sessions.backends.cache) is fast but seems essentially useless because cache entries can be evicted at any time, and it isn't very nice to require users to re-authenticate whenever that happens. * The file-based session store (django.contrib.sessions.backends.cache) would require a network file system when deployed on a site that uses multiple web servers. (I'm not sure about the relative performance of the file vs. cache backends, but I assume that memcached would be faster than using a network file system.) * The database-backed session store (django.contrib.sessions.backends.db) requires a database query every time you retrieve the session. And for some sites, an extra query can be prohibitive -- especially if it is the only query on the request, which requires setting up a DB connection. A cache+db session backend has pretty good performance in scenarios when sessions are frequently accessed but infrequently modified, since it basically acts like the cache backend except that when a session gets evicted in the cache it can still be retrieved from the DB. Of course that means that modifying the session is slower, since it needs to be saved to both the cache and the DB. But I think there are many use cases where the extra overhead on writes is not very significant. Thoughts? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Lazy-loading model fields
Hey Alex, On the trac ticket, I mentioned that my motivating reason for adding specifying default-lazy fields in the Field definition (i.e., without having to specify them on every query set) is the implicit queries that Django creates for foreign key lookups (e.g., item = obj.relateditem causes an implicit SQL query, and I want to avoid loading large text fields on the related item). Also, to a lesser extent, it would be good to avoid loading these columns in select_related(). I hadn't thought before about subclassing Query/QuerySet/Manager/ TextField to do this without changing the installed version of Django. That's an interesting idea. My change did also involve changing Options (i.e., _meta), though. Currently, however, the Query and QuerySet classes are functionally decomposed in a way that makes it hard to override most of this functionality in a subclass without copy-and-pasting a very large amount of code. (Compare, for instance, django.contrib.gis.db.models.sql.Query.get_columns with django.db.models.sql.Query.get_columns.) I just posted a patch to http://code.djangoproject.com/ticket/9368 which would make overriding this functionality somewhat easier. But it would still involve a considerable amount of code duplication if that logic wasn't in the Query base class. -Jesse On Oct 14, 10:00 am, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> wrote: > I think the sensible solution is to have what the ticket originally > suggested, a queryset/manager method to control whether a field is > lazy loading, and yes, that can be implemented without patching > django, you can, of course, make this the default by overiding the > default manager. > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Lazy-loading model fields
I think the sensible solution is to have what the ticket originally suggested, a queryset/manager method to control whether a field is lazy loading, and yes, that can be implemented without patching django, you can, of course, make this the default by overiding the default manager. On Oct 14, 12:56 pm, "Marty Alchin" <[EMAIL PROTECTED]> wrote: > On Tue, Oct 14, 2008 at 12:34 PM, Jesse Young <[EMAIL PROTECTED]> wrote: > > A LazyTextField that manages a dynamic model behind the scenes sounds > > like a neat idea. That would certainly make the Django side of this > > easier to manage. (But of course I have a slight bias towards code > > that exists already -- has anyone actually implemented this?) > > I don't think it's out there anywhere, but if you like I could whip up > the code for it in about 15 minutes (probably less). I just haven't > bothered because I don't personally need it. > > > Anyway, I'm only trying to point out that inline lazy fields are > > *sometimes* useful, and that Django shouldn't force people to do it by > > splitting out fields into separate tables. > > Damn it. Now you've gotten me thinking of ways to possibly manage a > field the way you want (one table, lazy loading, no data migration, no > patches to django, one-line change to the model). I'm 98% sure it's > possible, and I know of many of the things that would need to be > addressed to do so, but I really shouldn't be spending my time on it. > I probably won't be able to get it out of my head until I get it > working though, unfortunately. If I do write it down, I'll let you > know. > > Of course, it would also have to come with big biohazard signs warning > people that there could (and likely would) be unforeseen side-effects, > particularly in the admin, serializers, modelforms ... actually, > pretty much anything that introspects your model would never see that > field. I'm not sure if that's suitable for your case, but it certainly > wouldn't be good enough for wide distribution. > > -Gul --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Lazy-loading model fields
On Tue, Oct 14, 2008 at 12:34 PM, Jesse Young <[EMAIL PROTECTED]> wrote: > A LazyTextField that manages a dynamic model behind the scenes sounds > like a neat idea. That would certainly make the Django side of this > easier to manage. (But of course I have a slight bias towards code > that exists already -- has anyone actually implemented this?) I don't think it's out there anywhere, but if you like I could whip up the code for it in about 15 minutes (probably less). I just haven't bothered because I don't personally need it. > Anyway, I'm only trying to point out that inline lazy fields are > *sometimes* useful, and that Django shouldn't force people to do it by > splitting out fields into separate tables. Damn it. Now you've gotten me thinking of ways to possibly manage a field the way you want (one table, lazy loading, no data migration, no patches to django, one-line change to the model). I'm 98% sure it's possible, and I know of many of the things that would need to be addressed to do so, but I really shouldn't be spending my time on it. I probably won't be able to get it out of my head until I get it working though, unfortunately. If I do write it down, I'll let you know. Of course, it would also have to come with big biohazard signs warning people that there could (and likely would) be unforeseen side-effects, particularly in the admin, serializers, modelforms ... actually, pretty much anything that introspects your model would never see that field. I'm not sure if that's suitable for your case, but it certainly wouldn't be good enough for wide distribution. -Gul --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Lazy-loading model fields
> And LazyTextField could create the associated model behind the scenes, > set up a descriptor for loading its contents only when necessary, > cache those contents so they only get queried once, etc. And all this > without a single patch to Django's core and without you having to > manually maintain another model. A LazyTextField that manages a dynamic model behind the scenes sounds like a neat idea. That would certainly make the Django side of this easier to manage. (But of course I have a slight bias towards code that exists already -- has anyone actually implemented this?) > Admittedly, the relational approach would require a data migration, > but that's life sometimes. When an application is live and has a very large amount of data and users, avoiding data migration is extremely important. Data migration means that when we update our live website, someone has to run sql commands on our live database servers, which takes time and can add a lot of load; they have to reason about what would happen if someone visits our website if the sql is updated but not the python code (or vice versa), or whether they have to disable the website while the migration occurs. And the person who updates the website might be a different person than the developer who made the code change. So avoiding data migration just by making a one-line change in the Python code is a huge plus. Anyway, I'm only trying to point out that inline lazy fields are *sometimes* useful, and that Django shouldn't force people to do it by splitting out fields into separate tables. -Jesse --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Lazy-loading model fields
Moving discussion from http://code.djangoproject.com/ticket/5420. To summarize, I patched my local version of Django by adding a boolean 'lazy' parameter to the Field constructor, e.g. models.TextField(lazy=True), which determines whether Django loads that field by default on select queries, and also added one function to the manager and query set, toggle_fields(fetch=None, lazy=None, fetch_only=None), where each argument can be an array of field names (or None), to change the fields that would be retrieved by that particular query. mtredinnick replied: "You are implementing quite a different feature: fields that are always lazy loaded. That's a data modelling issue, not a retrieval issue (although pretty much this whole ticket is about working around data modelling that probably should be done differently in the first place, but c'est la vie). If you don't ever want them loaded with the main data, that suggests your data modelling has gone slightly awry and those fields shouldn't be part of the main model. Since foreign key relations are loaded lazily in the normal course of events, simply put the things you don't want loaded with the main model into another table and it will always lazy load upon access." I agree that it is possible to do a similar thing by split large fields into a separate table. However, there are also good reasons not to split them out into separate tables: * If you split them out, you have to manually create one model every time you create the other model. * If a fraction of your queries actually need those large fields, then you have to do a join, which can be expensive. * If your data access pattern changes after your application has actual data, it is much harder to migrate your data than to make a one line change to the application code (i.e., by adding lazy=True to the field definition). * Maintaining more tables is more work than maintaining fewer tables. Anyway, I don't really mind if Django decides it doesn't want to include this. We'll use it anyway. Just trying to help. -Jesse --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Lazy-loading model fields
I won't pretend to know the whole story here, but I will take issue with some of your points. I'll leave others to people who have history on the subject. On Tue, Oct 14, 2008 at 3:14 AM, Jesse Young <[EMAIL PROTECTED]> wrote: > * If you split them out, you have to manually create one model every > time you create the other model. If by "manually create one model" you mean "type out a brand new model definition in Python" you're flat-out wrong. I've done quite a bit of work with generating models dynamically[1] and what you're asking for could be made very simple using that approach. Rather than this: thesis = models.TextField(lazy=True) you could use this instead: thesis = lazy_utils.LazyTextField() And LazyTextField could create the associated model behind the scenes, set up a descriptor for loading its contents only when necessary, cache those contents so they only get queried once, etc. And all this without a single patch to Django's core and without you having to manually maintain another model. > * If a fraction of your queries actually need those large fields, then > you have to do a join, which can be expensive. It doesn't address this, of course, but like I said, I'll leave that to the professionals. > * If your data access pattern changes after your application has > actual data, it is much harder to migrate your data than to make a one > line change to the application code (i.e., by adding lazy=True to the > field definition). Admittedly, the relational approach would require a data migration, but that's life sometimes. There are plenty of tools now to make changes like this relatively easy to deal with, and I expect a custom LazyTextField could be written to give extra hints to any of them. > * Maintaining more tables is more work than maintaining fewer tables. Again, this depends on what you mean by "maintaining". At a raw database level, yes, that's true, but if you're talking about the Python level, you can make that extra table almost non-existent and work with it as a regular field. Anyway, I don't expect this information to solve your problem, but I hope I've cleared up a few inaccuracies in your justification. Maybe it'll help you with whatever solution you do come up with. -Gul [1] http://code.djangoproject.com/wiki/DynamicModels --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
#6845 - model validation status
Hi,
I took a great email from Gary Wilson to get me started on the status structure.
Some of the core design decisions were finalised by Jacob, Malcolm
(please scream if I have misunderstood or don't remember things
correctly) and me at DjangoCon.
Honza Král
E-Mail: [EMAIL PROTECTED]
ICQ#: 107471613
Phone: +420 606 678585
* The ValidationError class has moved from forms.utils to
django.core.exceptions now that it is used for more than just forms.
ErrorList has been kept in forms.utils
validators
==
not yet implemented - talking about plans here
* New validators (replacements for many, but not all of the old validators)
have been put in ``django.core.validators``
* the new validator is a function like:
def validate_something( value, all_values={}, instance=None ):
if i_dont_like( value ):
raise ValidationError( 'I dont like %r' % value,
error_code=validators.SOME_CODE )
the error_code can be used later in Forms to look up custom error messages.
django.db.models.Model class
* validate_unique has been moved here from ModelForm
* clean and validate methods were added, the idea is that any
__validation__ should never alter the data, that's why the top-level
function that can result in data change is clean() and not validate()
* clean() calls clean() on individual fields and then calls
validate(), ValidationError with message_dict containing the errors is
raised when errors occur
* validate() calls validate_unique() and is a hook to add custom
instance-wide validation (just raise ValidationError from it)
* no validate_FIELD or clean_FIELD hooks on models
* validate is run even if some errors occur on the fields' clean() methods
Model fields (django.db.models.fields)
==
* clean and validate methods were added, see Model for function
Form Fields (django.forms.fields)
=
* form fields haven't been changed yet, I will probably try and do
the same thinng the old patch did:
* The to_python() method remains and a validate() method have been added.
Fields have had their clean() methods split into to_python() and validate()
methods.
* clean() now calls to_python() then validate().
django.forms.forms.BaseForm class
=
* no changes here either, so far. Will probably also try and do:
* Form gets a new validate() method, which is meant to replace clean(), and
full_clean() now looks for validate_ methods, which are meant to
replace clean_ methods. full_clean() also uses the Form.validate()
instead of Form.clean() to set the cleaned_data.
- The validate_ methods are passed the Field.clean()'ed value (yea
for fixing #3896).
django.forms.models.BaseModelForm class
===
* save_instance has been split into two functions:
* make_instance(form, instance, fields=None, exclude=None) which
return unsaved instance with populated fields
* save_maked_instance(form, instance, fields=None, commit=True,
fail_message='saved') which saves the populated instance (and it's m2m
data)
save_instance remains and calls the two new functions
* in it's clean() method constructs the instance with make_instance,
then runs validation on it
* the validate_unique method has been moved to Model
* the form's save() method only calls save_maked_instance
django.forms.models.BaseModelFormset class
===
* calls form.save() in save_new and save_existing
known Issues
=
* InlineFormSet.save_as_new is broken
unknown Issues
==
* I am fairly sure there are many since I only wrote minimal tests
I would appreciate any feedback on this, especially on:
should we change the current Form validation? - split to_python and
validate, etc..
how big an effort should we make to get all the validation to one
place - the validators (ie remove the logic from FormFields and just
have them call the validators where possible)
testing - anybody willing to write tests and/or have a piece of code
that I broke, please contact me
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---
Re: auto_now_add and auto_now
On 13 oct, 17:15, Brian Beck <[EMAIL PROTECTED]> wrote: > On Oct 13, 8:38 am, "Mike Scott" <[EMAIL PROTECTED]> wrote: > > > Secondly this question has been asked, and solved many times. If you search > > through the django-users archives I'm sure you'll find plenty of solutions. > > There are solutions out and about in the blogosphere too. > > Actually I think this is an appropriate place to ask since the > resolution (which was unclear) never resulted in action. I was > confused about these myself just yesterday because the documentation > doesn't mention anything about possible future deprecation. The only > action was three months ago with a DeprecationWarning patch that isn't > checked in. Thanks Brian, that why I ask here, I was really confused. Thierry. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Static vs variable test fixtures
After `testFOO` method call test case flushes all data in database. So if you want to add some logic at this stage - flushing procedure may take a lot of time and you will get no speed improvements. On Oct 14, 5:18 am, Simon Litchfield <[EMAIL PROTECTED]> wrote: > Reloading of fixtures for each test gets painful when the fixtures get > big. > > Maybe we could continue to have TestCase.fixtures as a list of > 'variable' fixtures that do definitely need to be reloaded for each > test... and add a new optional attribute, TestCase.static_fixtures, > which would be a list of fixtures that only need to be loaded at the > start of the TestCase. > > I've run into this problem a few times now and this simple improvement > would make tests so much nicer :-) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
