Hi Malcolm, Thank you VERY much for taking the time to explain this to me. It makes perfect sense and since yesterday I've delved a bit deeper (than ever) into the django code itself to understand a little more. You are ABSOLUTELY right to point out that I'm in user-beware country. I don't apologize for that so much. I think the batchimport project has some usefulness to folks and it's worth the risk -- **IF** I do it right. ;-) (if you think otherwise, PLEASE let me know)
Even since yesterday I've begun a good scouring of the project to reduce touching the internals as much as possible. I apologize for not giving more details. I am always trying to balance giving enough information while trying to avoid scaring people away with too much detail. I didn't think it through enough I suppose. I kept being stymied by the thing "working" on the command line. My mistake. Anyway. I understand the whole thing MUCH better now and have already spent a little time reviewing the really "cutting edge" aspects of this project like, you know, "how lists work in python." ;-) Ugh. Kicking Myself Diligently, Keyton On Jan 22, 3:10 am, Malcolm Tredinnick <malc...@pointy-stick.com> wrote: > On Wed, 2009-01-21 at 19:38 -0800,KeytonWeissinger wrote: > > OK. I figured it out and it was VERY STRANGE. Maybe some one reading > > this can tell me WWWHHHYYY my solution works. For now, it's voodoo to > > me... > > You're going to kick yourself after this. The reason for the failure > turns out to be really easy to understand, now that your show the > code. :-) > > > > > Turns out that some of my code (in the view where I was having > > trouble) called some code that looked sort of like this: > > Hmm .. you didn't exactly simplify your failing case to the smallest > possible example, did you? All this extra stuff would have set off a lot > of alarm bells about other places to look (although my first thought > would have been whether you could do without it and retest to see if > that changed anything). > > Before getting into the more-or-less two line explanation, some > background information. Firstly, model._meta is obviously internal API, > so there are some shortcuts taken for performance and maintainability > reasons. We don't send the black helicopters after people who access it, > but it's very much "user beware" country. It's one of the areas that is > currently intentionally undocumented, for example (we'll eventually drop > some documentation about internal datastructures into docs/internals/, > but have had other things to do so far). > > Of relevance to this particular case, we use lists a lot internally and > return references to them from certain methods. We use lists, rather > than tuples, because they are updated a fair bit during creation and all > the extra copying isn't free (updating a tuple requires a copy). Lists > are just nicer to work with, generally. > > > field_list = model._meta.fields > > Have a look at the implementation here > (django.db.models.options.Options._fields(), since _meta.fields is a > property). It returns a reference to a list. A *reference*, not a copy! > > Your code would have worked if you wrote > > # Work with a copy of the fields list. > field_list = model._meta.fields[:] > > > field_list.extend(model._meta.many_to_many) > > Here you update that reference in place. So you've just corrupted one of > the internal data structures of self._meta. > > That's pretty much going to guarantee that the wheels will fall off at > some point in the future. If not in the way you saw, in some other > really hard to debug fashion. > > It's kind of a Python trap and if you're designing a public API, you > would generally return a copy of the list if the caller wasn't expected > to change the internal structure. In the Options class, we "know" (in > quotes, because there are people who call it from public code, but > that's their problem, not ours) that the only callers are functions that > are read-only on the data structure, so we save a bit of time and code > complexity. We could return a copy each time, but we access self._fields > a *lot*, particularly in queryset operations. Things might have changed > a bit in the last six months (although not dramatically), but when I was > working on the queryset-refactor branch, I spent a lot of time profiling > various common execution paths and a lot of the newer shared data > structures in Options are implemented based on that work (self.fields > used to be a simple list, for example, but when model inheritance came > in, it became, internally, more complicated). > > Regards, > Malcolm --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~----------~----~----~----~------~----~------~--~---
