Re: Thoughts on solution to forward references in MySQL (#3615)
Hi Russell and Jim, On 07/06/2011 05:34 PM, Russell Keith-Magee wrote: > On Thu, Jul 7, 2011 at 5:05 AM, Jim D.wrote: >> * There's a DB feature can_defer_constraint_checks . I couldn't find much >> by way of documentation or or usage of this feature. But I was trying to >> figure out if the work we are doing here is what this feature refers to, and >> if so, if we should be marking this as True for MySQL and SQLite with this >> implementation. I'm not sure there is other behavior that is required or >> expected in that attribute. Anyhow, that might also be a path forward for >> the issue I raise above (ie. we could skip the test if >> can_defer_constraint_checks is not True). > > This feature flag exists essentially to verify whether MySQL InnoDB is > currently in use. It isn't used during normal runtime; it's purely a > test skipping flag. It's used to identify tests that need to be > skipped because the test data requires a circular reference which > (historically) hasn't been possible under InnoDB because constraints > aren't checked. > > Essentially, this feature flag shouldn't need to exist at all; it only > exists so that InnoDB passes the test suite without errors. If we are > able to resolve the issue that allows MySQL to use forward references > in data, then this feature flag shouldn't be required at all. This isn't true since the introduction of the new deletion code last fall. The feature flag is used in db/models/deletion.py to detect whether it is necessary to null out nullable FKs prior to deletion. On backends which can defer constraints, this nulling-out would add a needless extra query. On backends which can't defer, nulling-out means the nullable FK relationship doesn't dictate a deletion-ordering constraint, which lets us handle circular FK references as long as at least one of them is nullable (otherwise we'd have to just throw up our hands and hope for the best, likely resulting in an integrity error). Since this use of the flag is unrelated to fixture loading, the fixture-loading fix should not change the value of the flag for any backend. Carl -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Thoughts on solution to forward references in MySQL (#3615)
If you aren't using InnoDB, then it probably doesn't matter if you turn foreign key checks on or off: it becomes a no-op. There are some other storage engines that support transactions, and some of them might "do the right thing" with respect to deferred foreign key checks, but I think it does no harm to disable the checks on loading fixtures on these engines. In other words, always turn them off when loading fixtures (and back on afterwards) and don't care about the storage engine and it should be fine. Keep it simple. On Jul 6, 2011 7:34 PM, "Russell Keith-Magee"wrote: > > On Thu, Jul 7, 2011 at 5:05 AM, Jim D. wrote: > > Are any core devs interested in taking a closer look at the current patch on > > this ticket (https://code.djangoproject.com/ticket/3615)? It's been through > > several rounds of revision after discussion here and on the ticket comment > > thread. > > Interested - most certainly. The issue (as always) is finding the time :-) > > I'm certainly grateful for your efforts; this is a long standing issue > that I'd love to see addressed; I'm just not in a position to commit > to anything. I'll put this on my list of things to look at, but I'm > not getting a whole lot of time to look at that list at the moment. > > > * There's a DB feature can_defer_constraint_checks . I couldn't find much > > by way of documentation or or usage of this feature. But I was trying to > > figure out if the work we are doing here is what this feature refers to, and > > if so, if we should be marking this as True for MySQL and SQLite with this > > implementation. I'm not sure there is other behavior that is required or > > expected in that attribute. Anyhow, that might also be a path forward for > > the issue I raise above (ie. we could skip the test if > > can_defer_constraint_checks is not True). > > This feature flag exists essentially to verify whether MySQL InnoDB is > currently in use. It isn't used during normal runtime; it's purely a > test skipping flag. It's used to identify tests that need to be > skipped because the test data requires a circular reference which > (historically) hasn't been possible under InnoDB because constraints > aren't checked. > > Essentially, this feature flag shouldn't need to exist at all; it only > exists so that InnoDB passes the test suite without errors. If we are > able to resolve the issue that allows MySQL to use forward references > in data, then this feature flag shouldn't be required at all. > > Yours, > Russ Magee %-) > > -- > You received this message because you are subscribed to the Google Groups "Django developers" group. > To post to this group, send email to django-developers@googlegroups.com. > To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/django-developers?hl=en. > -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: [GSOC] Template Engine Compilation and Runtime Refactoring Progress
Hi, Status of this week: I spend the last week mostly trying to figure out how to debug a bunch of annoying issues and due to the slow progress I decided on doing an AST -> Python code translation for debugging purposes. The identifier tracking method I naively started using initially (which is assuming that there are no branches and recording identifiers for each scope while the transformer is creating code) turns out to be by far not good enough as it misses identifiers that are referenced after blocks. I will need to heavily improve on that. Regards, Armin -- You received this message because you are subscribed to the Google Groups "Django developers" group. To view this discussion on the web visit https://groups.google.com/d/msg/django-developers/-/mITAVQia2fEJ. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Thoughts on solution to forward references in MySQL (#3615)
On Thu, Jul 7, 2011 at 5:05 AM, Jim D.wrote: > Are any core devs interested in taking a closer look at the current patch on > this ticket (https://code.djangoproject.com/ticket/3615)? It's been through > several rounds of revision after discussion here and on the ticket comment > thread. Interested - most certainly. The issue (as always) is finding the time :-) I'm certainly grateful for your efforts; this is a long standing issue that I'd love to see addressed; I'm just not in a position to commit to anything. I'll put this on my list of things to look at, but I'm not getting a whole lot of time to look at that list at the moment. > * There's a DB feature can_defer_constraint_checks . I couldn't find much > by way of documentation or or usage of this feature. But I was trying to > figure out if the work we are doing here is what this feature refers to, and > if so, if we should be marking this as True for MySQL and SQLite with this > implementation. I'm not sure there is other behavior that is required or > expected in that attribute. Anyhow, that might also be a path forward for > the issue I raise above (ie. we could skip the test if > can_defer_constraint_checks is not True). This feature flag exists essentially to verify whether MySQL InnoDB is currently in use. It isn't used during normal runtime; it's purely a test skipping flag. It's used to identify tests that need to be skipped because the test data requires a circular reference which (historically) hasn't been possible under InnoDB because constraints aren't checked. Essentially, this feature flag shouldn't need to exist at all; it only exists so that InnoDB passes the test suite without errors. If we are able to resolve the issue that allows MySQL to use forward references in data, then this feature flag shouldn't be required at all. Yours, Russ Magee %-) -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: FilePathField and stale choices
On 06/07/11 21:41, Daniel Swarbrick wrote: > Or can we perhaps even add an option to FilePathField that would > refresh the choices each time? I'm not entirely sure how that would > work... proxy function maybe? That sounds like a reasonable idea to me, since doing it by default could impose a serious performance hit. In it's most naive form (i.e. making the 'choices' attribute a lazy list using django.utils.functional.lazy) this wouldn't be too hard to implement. However, that would mean that *every* time it was accessed it would have to walk the file system to get the choices, which could easily get crazy. A better solution would add some level of caching. This sounds like we need an option that specifies the number of seconds to cache for. It would be OK to require that there is a cache backend active for this to work. We could have a single setting that controls the two e.g. cache_paths_for - if it was None (the default) then it would cache forever. Feel free to open a ticket for this. Luke -- A mosquito cried out in pain: "A chemist has poisoned my brain!" The cause of his sorrow was para-dichloro- diphenyltrichloroethane Luke Plant || http://lukeplant.me.uk/ -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Thoughts on solution to forward references in MySQL (#3615)
Are any core devs interested in taking a closer look at the current patch on this ticket (https://code.djangoproject.com/ticket/3615)? It's been through several rounds of revision after discussion here and on the ticket comment thread. As of right now the patch should apply cleanly and pass tests in fixtures_regress. It solves the problem of IntegrityError exceptions when loading a fixture in MySQL, and it also applies foreign key constraint checks for both MySQL and SQLite to ensure valid data is being loaded. Last, it offers two hooks for other backends to implement constraint checks after fixtures are loaded. The issues I still had questions on: * There is a test error in fixtures_regress.TestFixtures.test_loaddata_raises_error_when_fixture_has_invalid_foreign_key() if a backend does not implement a solution to check for foreign keys. Basically I have MySQL and SQlite covered, but not postgresql or Oracle. As I mention in the ticket, the work done here https://code.djangoproject.com/ticket/11665 covers Postgresql, so all it needs is an implementation from an Oracle user. * There's a DB feature can_defer_constraint_checks . I couldn't find much by way of documentation or or usage of this feature. But I was trying to figure out if the work we are doing here is what this feature refers to, and if so, if we should be marking this as True for MySQL and SQLite with this implementation. I'm not sure there is other behavior that is required or expected in that attribute. Anyhow, that might also be a path forward for the issue I raise above (ie. we could skip the test if can_defer_constraint_checks is not True). I know there is other very similar work being coordinated elsewhere, so hopefully someone with an an eye on the big picture can help me see this through the last mile. Aside from the minor details above, this should be ready for checkin. Thanks! -- You received this message because you are subscribed to the Google Groups "Django developers" group. To view this discussion on the web visit https://groups.google.com/d/msg/django-developers/-/q_RzrPrYHboJ. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Blank page returned on POST if csrf middleware is not configured... bug?
Hi Luke, Strange, I performed a bare test and I can't seem to reproduce the problem. I then tried to replicate the problem in the existing code, and it doesn't happen any more. Some changes were being made earlier today to our L7 traffic inspection appliance, so I suspect this might have been the reason for the blank result on POST (perhaps I hit it in the middle of a change being made to the appliance rules). Always annoying to not know 100% what caused a bug :X Thanks for the quick response anyway, in future I'll make sure to test on a fresh install before posting to django-developers. Cal On Wed, Jul 6, 2011 at 7:22 PM, Cal Leeming [Simplicity Media Ltd]wrote: > Ah, okay I'll do it on both 1.2 and 1.3 to determine if it is/was a bug. > > Cal > > On Wed, Jul 6, 2011 at 7:18 PM, Luke Plant wrote: >> On 06/07/11 18:12, Cal Leeming [Simplicity Media Ltd] wrote: >>> Hi Luke, >>> >>> Thanks for the reply. >>> >>> I'll set up a test case in a fresh 1.2 django instance, and will let >>> you know the results (and the code used). >> >> If it's not present in 1.3/trunk, there won't be bug fixes for it >> (unless it is security related). >> >> Luke >> >> -- >> A mosquito cried out in pain: >> "A chemist has poisoned my brain!" >> The cause of his sorrow >> was para-dichloro- >> diphenyltrichloroethane >> >> Luke Plant || http://lukeplant.me.uk/ >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Django developers" group. >> To post to this group, send email to django-developers@googlegroups.com. >> To unsubscribe from this group, send email to >> django-developers+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/django-developers?hl=en. >> >> > -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Blank page returned on POST if csrf middleware is not configured... bug?
Ah, okay I'll do it on both 1.2 and 1.3 to determine if it is/was a bug. Cal On Wed, Jul 6, 2011 at 7:18 PM, Luke Plantwrote: > On 06/07/11 18:12, Cal Leeming [Simplicity Media Ltd] wrote: >> Hi Luke, >> >> Thanks for the reply. >> >> I'll set up a test case in a fresh 1.2 django instance, and will let >> you know the results (and the code used). > > If it's not present in 1.3/trunk, there won't be bug fixes for it > (unless it is security related). > > Luke > > -- > A mosquito cried out in pain: > "A chemist has poisoned my brain!" > The cause of his sorrow > was para-dichloro- > diphenyltrichloroethane > > Luke Plant || http://lukeplant.me.uk/ > > -- > You received this message because you are subscribed to the Google Groups > "Django developers" group. > To post to this group, send email to django-developers@googlegroups.com. > To unsubscribe from this group, send email to > django-developers+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/django-developers?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Blank page returned on POST if csrf middleware is not configured... bug?
On 06/07/11 18:12, Cal Leeming [Simplicity Media Ltd] wrote: > Hi Luke, > > Thanks for the reply. > > I'll set up a test case in a fresh 1.2 django instance, and will let > you know the results (and the code used). If it's not present in 1.3/trunk, there won't be bug fixes for it (unless it is security related). Luke -- A mosquito cried out in pain: "A chemist has poisoned my brain!" The cause of his sorrow was para-dichloro- diphenyltrichloroethane Luke Plant || http://lukeplant.me.uk/ -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Blank page returned on POST if csrf middleware is not configured... bug?
Hi Luke, Thanks for the reply. I'll set up a test case in a fresh 1.2 django instance, and will let you know the results (and the code used). Cal On Wed, Jul 6, 2011 at 5:41 PM, Luke Plantwrote: > On 06/07/11 15:43, Cal Leeming [Simplicity Media Ltd] wrote: >> Today I came across a very strange problem. >> >> When attempting to post a form (without the CSRF middleware being >> present in MIDDLEWARE_CLASSES), django would *always* return a blank >> page. If the post contained no data, it would come back fine, but if >> it did contain data, it would come back blank. >> >> Upon adding the CSRF middleware into the MIDDLEWARE_CLASSES, and >> adding {% csrf_token %} into the template, it would return back fine >> (and without csrf_token - it would return back an exception, as it >> should). >> >> Although it is a good thing that it happened (because I did actually >> forget to include CSRF middleware), I feel that Django should raise an >> exception (if this is the default behaviour), to tell the user they >> must enable CSRF middleware, rather than just returning a blank page. >> >> Any thoughts?? > > Sgreed, but there isn't enough information here to work out what is > going on. Please file a ticket if you can create a test case that will > reproduce it. I can't think of any code paths in Django that would > return 'a blank page' (though I'm not sure I know what you mean by that). > > Luke > > > -- > A mosquito cried out in pain: > "A chemist has poisoned my brain!" > The cause of his sorrow > was para-dichloro- > diphenyltrichloroethane > > Luke Plant || http://lukeplant.me.uk/ > > -- > You received this message because you are subscribed to the Google Groups > "Django developers" group. > To post to this group, send email to django-developers@googlegroups.com. > To unsubscribe from this group, send email to > django-developers+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/django-developers?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Blank page returned on POST if csrf middleware is not configured... bug?
On 06/07/11 15:43, Cal Leeming [Simplicity Media Ltd] wrote: > Today I came across a very strange problem. > > When attempting to post a form (without the CSRF middleware being > present in MIDDLEWARE_CLASSES), django would *always* return a blank > page. If the post contained no data, it would come back fine, but if > it did contain data, it would come back blank. > > Upon adding the CSRF middleware into the MIDDLEWARE_CLASSES, and > adding {% csrf_token %} into the template, it would return back fine > (and without csrf_token - it would return back an exception, as it > should). > > Although it is a good thing that it happened (because I did actually > forget to include CSRF middleware), I feel that Django should raise an > exception (if this is the default behaviour), to tell the user they > must enable CSRF middleware, rather than just returning a blank page. > > Any thoughts?? Sgreed, but there isn't enough information here to work out what is going on. Please file a ticket if you can create a test case that will reproduce it. I can't think of any code paths in Django that would return 'a blank page' (though I'm not sure I know what you mean by that). Luke -- A mosquito cried out in pain: "A chemist has poisoned my brain!" The cause of his sorrow was para-dichloro- diphenyltrichloroethane Luke Plant || http://lukeplant.me.uk/ -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Blank page returned on POST if csrf middleware is not configured... bug?
Today I came across a very strange problem. When attempting to post a form (without the CSRF middleware being present in MIDDLEWARE_CLASSES), django would *always* return a blank page. If the post contained no data, it would come back fine, but if it did contain data, it would come back blank. Upon adding the CSRF middleware into the MIDDLEWARE_CLASSES, and adding {% csrf_token %} into the template, it would return back fine (and without csrf_token - it would return back an exception, as it should). Although it is a good thing that it happened (because I did actually forget to include CSRF middleware), I feel that Django should raise an exception (if this is the default behaviour), to tell the user they must enable CSRF middleware, rather than just returning a blank page. Any thoughts?? Cal -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Weekly check-in (this should be #5, right...?)
On Tue, Jul 05, 2011 at 03:39:53PM -0700, akaariai wrote: > I did a glance-over of your github branch. I was especially looking > for how you will handle LEFT OUTER JOINS involving composite primary > keys / foreign keys. If I am not missing something, I think this > haven't been done yet. I have myself been thinking about this issue, > and I thought it would be good to share what I have found out. This is something I'd like to leave for the second half of the program. For now I'm focusing on making CompositeFields work in the model they've been added to, relationship fields and all this stuff is scheduled for the second half of July. (-: > The problematic part for multi-column join conditions is in > django.db.models.sql.query: > def join(self, connector, ...): > """ > Returns an alias for the join in 'connection', either reusing an > existing alias for that join or creating a new one. 'connection' > is a > tuple (lhs, table, lhs_col, col) where 'lhs' is either an existing > table alias or a table name. The join correspods to the SQL > equivalent > of:: > > lhs.lhs_col = table.col > """ > > Obviously this can not work for creating multi-column joins. > > The connection information is stored in alias_map, join_map and > rev_join_map. In particular, in alias_map is stored (table, alias, > join_type, lhs, lhs_col, col, nullable). Currently the contents of the > alias_map is turned into SQL (sql/compliler.py, get_from_clause()) as: > > result.append('%s %s%s ON (%s.%s = %s.%s)' > % (join_type, qn(name), alias_str, qn(lhs), > qn2(lhs_col), qn(alias), qn2(col))) > > The most simple way to extend this to contain more columns would > probably be the following: > - connection is defined as (lhs, table, lhs_col1, col1, lhs_col2, > col2, ...) > - alias_map format needs to change a bit so that the extra columns > can be stored in there. One could store the extra column after the > nullable. Cleaner would be to have the columns in one tuple: (table, > alias, join_type, lhs, (cols), nullable) > - Limited amount of places needs to be fixed, most notably the > get_from_clause() of compiler.py > > The downside of the above is that it does not support any other join > conditions than ones involving 2 tables and a list of anded columns. > For composite fields this is enough. Yeah, I had something like this planned. To be precise, the exact same thing you wrote in the second e-mail. > For future usage it would be nice if one could pass in Where nodes as > the connection. This would allow for arbitrary join conditions. The > Where node knows how to turn itself into SQL, how to relabel aliases > and so on. This approach has some problems, however: > - How to generate the Where node? > - How to match existing joins to new joins? Currently this is done > by checking that the connection four-tuple is equivalent to the > existing join's four tuple. I don't think Where nodes know how to > check equivalence to another node. And even if where nodes knew how to > do that, also all the leaf nodes would need to know how to do that. > - Performance issues, cloning a Where node is more expensive than > cloning a tuple. Also construction, equivalence checking and other > operations too are somewhat more expensive than using tuples. > - Overkill for composite fields > > Of course, the approaches could be combined, that is you pass in the > join condition as a tuple, and you can pass extra_filters (default > None) as a Where node. This would keep the normal case efficient but > allow for more complex join conditions if really needed. The join > having extra_filters could not be reused, except when explicitly > stated. I don't know, somehow I'm not entirely convinced this would be really useful. Yes, I realize sometimes people need more complex joins etc, but that's what raw queries are for. Besides, I can't really imagine how a user might use this feature -- I don't see how a QuerySet filter could be used to supply these extra conditions, only if we added some hooks things like custom fields could use or something like that. But still, this would add a ton of complexity into the code for little gain, IMHO. Anyway, should there be consensus that we want something like this, I'll gladly look into it sometime later. But for now, to keep things realistic as far as my GSoC project is concerned, I'll stick to the easier solution. > Having said all this, for this project "extend the connection tuple" > approach seems to be the only sane choice. > > The work you have done looks very promising. I hope this post has been > at least somewhat useful to you. Thanks, I really appreciate it. On Tue, Jul 05, 2011 at 06:15:50PM -0700, akaariai wrote: > I implemented this in my github branch > https://github.com/akaariai/django/tree/composite_join > > With this you can do: > a = Article.objects.all() > a.query.get_initial_alias() > a.query.join(('basic_article', 'foo', ('pk_part1',
Problem with reversing namespaced URLs by a view name
Hi, I have encountered reverse() behavior I believe to be invalid. In my separate app's URLconf I have included a pattern pointing to some contrib.auth view. This app's URLconf I've included in my main project's URLconf and gave it a namespace. Now an other contrib.auth view cannot reverse an URL to the first one using only a view name. If I take that namespace out everything works like a charm. Shouldn't namespaces only affect reversing by a url name? The question is -- a bug or a feature? -- You received this message because you are subscribed to the Google Groups "Django developers" group. To view this discussion on the web visit https://groups.google.com/d/msg/django-developers/-/e5f_PUzBqu0J. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.