[Zope-dev] funky side-effects, possible bug in HTTPRequest.py
The last few days I've been working on patch to Zope 2 that will clean up the textarea size preference handling in the ZMI. Right now, its a mess. I kept running into this really irritating behavior such that, when ever I'd go to pull something out of a request object, it'd end up being found in the 'other' dictionary instead of where I expected to be (which happened to be 'form'). After getting really curious as to why this was happening I've managed to track it down to subtle a change in the way that the HTTPRequest.get method worked between revisions 1.75 and 1.74 (1.75 being mj's merge of the value tainting code). The code responsible (assuming the value isn't tainted): # Untrusted data *after* trusted data v = self.form.get(key, _marker) if v is not _marker: other[key] = v # *boom* return v That magical promotion of the key & value to the other dictionary is what tripped me up. This technique, while originally used only for known convenience variables like URLx or BASEx and for promoting lazy data, now applies to all variables after revision 1.75. (This isn't the only spot it gets used now, the tainting changes made it affect form values, cookies, tainted form values, etc.) It would increase the speed at which variables are found--but I'm not sure its really an intended side effect, and now I'll explain how I was bitten by it. Its not unusual to see people write methods to be published that are similar too: manage_main = DTMLFile("edit", globals()) def manage_changeFoo(self, REQUEST, something=None, or_other=628): # stuff return self.manage_main(self, REQUEST) They're a dime a dozen, yay neat. What sometimes makes them more interesting is when people modify the REQUEST dictionaries to pull off various behind the scenes fake-like-we-requested-it-that-way stuff. This is what several of the methods that handled the Taller, Wider, Narrower, etc. buttons on text area prefs did, and thats why I ran into it. One of these methods did: REQUEST.form.update({'something':'x', 'or_other':'y'}) and then it returned a DTMLFile plied with the snazzy new request that had been tricked out with these fake values now stuck into the form dictionary. This explains the TALES I saw that was "request/form/something | request/something | default" which when I first saw made wonder if the author was on acid. It was clear to me that 'something' would never be in other, it wasn't a special variable and it wasn't being explicitly stuck into the other dictionary by the code, so why the redundant TALES I wondered? I removed the first statement, and it broke the functionality. Hmmm! Until about 20 minutes ago I didn't understand exactly how object methods got published, but I tracked down the code in Publish.py and found the mapply() function and now its all clear to me what was happening. The request object gets mapply'd to the published method, this means that any positional arguments or any keyword arguments will get HTTPRequest.get'd out of the request object when they are applied to the published method, and thanks to the side-effects from revision 1.75 on, it also means any named variables in a method definition will be promoted into the HTTPRequest.other dictionary. Now, let me just say - if thats how its supposed to be, so be it - but spin my nipple nuts and call me Frank, this does NOT strike me as terribly intuitive behavior. Anyway, it was a learning experience for me, but I'm not convinced this isn't a bug. What do you think? (the patches I spoke of should be ready sometime tomorrowish assuming I don't run into any other funkyness, I'll post them to the collector) -- Jamie Heilman http://audible.transient.net/~jamie/ "We must be born with an intuition of mortality. Before we know the words for it, before we know there are words, out we come bloodied and squalling with the knowledge that for all the compasses in the world, there's only one direction, and time is its only measure." -Rosencrantz ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] funky side-effects, possible bug in HTTPRequest.py
On Fri, Jun 20, 2003 at 03:07:18AM -0700, Jamie Heilman wrote: > Anyway, it was a learning experience for me, but I'm not convinced > this isn't a bug. What do you think? >From the POV of a poor web monkey, my opinion is that this is purely awful. It's the kind of thing that can easily waste hours of debug time. I was just saying to Andy McKay yesterday that sometimes I think zope 2 is designed around the principle of greatest consternation rather than the principle of least suprise. Case in point. -- Paul Winkler http://www.slinkp.com Look! Up in the sky! It's ANTHROPOLOGIST MAN! (random hero from isometric.spaceninja.com) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] funky side-effects, possible bug in HTTPRequest.py
Jamie Heilman wrote: [major snippage] Hmmm, that means that this changes break exactly these applications, which, in order to be on the secure side, explicitly use REQUEST.form['bla'] more than once in a request, right. Ironic. cheers, oliver ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] funky side-effects, possible bug in HTTPRequest.py
Jamie Heilman wrote at 2003-6-20 03:07 -0700: > ... > That magical promotion of the key & value to the other dictionary is > what tripped me up. This technique, while originally used only for > known convenience variables like URLx or BASEx and for promoting lazy > data, now applies to all variables after revision 1.75. Zope promotes the variables from "cookies" and "form" to "other" at least since Zope 2.1.6 (the first version I worked with). I think this is for efficiency reasons. You have a single dictionary lookup instead of three ("other", "form", "cookie"). Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] funky side-effects, possible bug in HTTPRequest.py
Oliver Bleutgen wrote: > Jamie Heilman wrote: > >[major snippage] > > Hmmm, that means that this changes break exactly these applications, > which, in order to be on the secure side, explicitly use > REQUEST.form['bla'] more than once in a request, right. Naw, it doesn't break them, promotion doesn't mean the vars are actually moved from the form dictionary to the other dictionary, only copied. What it does is humble the poor shmuck who thought he knew the dangers of REQUEST.get well enough to use it without shooting himself in the foot. -- Jamie Heilman http://audible.transient.net/~jamie/ "You came all this way, without saying squat, and now you're trying to tell me a '56 Chevy can beat a '47 Buick in a dead quarter mile? I liked you better when you weren't saying squat kid." -Buddy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] funky side-effects, possible bug in HTTPRequest.py
Dieter Maurer wrote: > Zope promotes the variables from "cookies" and "form" to "other" > at least since Zope 2.1.6 (the first version I worked with). > > I think this is for efficiency reasons. You have a single > dictionary lookup instead of three ("other", "form", "cookie"). Yeah most promotion is for efficiency, but since 2.1.6 eh? Interesting, that means its happening elsewhere too. Hmm. Yeah, I see, it happens in __init__ for cookies, mmm and in processInputs for form variables I guess. The difference I can see is two fold, first, before 1.75 I believe promotion of form and cookies was only done during request object creation, interestingly, its not done there anymore from the looks of it; cookies and form vars are kept in their seperate tainted & non-tainted buckets until they are fetched. Second the promotion of cookies wouldn't clobber pre-existing keys in other, interestingly, in 2.1.6 times, form vars would clober normal other vars (URLx, BASEx, etc.). I must admit, this isn't the biggest deal to me, I just blew quite a bit of time tracking it down because I couldn't understand why the promotion was happening and I don't like surprises, and nothing about "def foo(self, bar, baz):" immediately jumped out and said to me, "promote bar and baz form vars and cookies to the other dictionary." Now that I have an appreciation for exactly how methods are published, I see things in a different light of course. -- Jamie Heilman http://audible.transient.net/~jamie/ "...thats the metaphorical equivalent of flopping your wedding tackle into a lion's mouth and flicking his lovespuds with a wet towel, pure insanity..." -Rimmer ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] funky side-effects, possible bug in HTTPRequest.py
In article <[EMAIL PROTECTED]> you write: > > # Untrusted data *after* trusted data > v = self.form.get(key, _marker) > if v is not _marker: > other[key] = v # *boom* > return v > > That magical promotion of the key & value to the other dictionary is > what tripped me up. Wouldn't other.setdefault(key, v) be better? So a variable already existing in other wouldn't get clobbered. Florent -- Florent Guillaume, Nuxeo (Paris, France) +33 1 40 33 79 87 http://nuxeo.com mailto:[EMAIL PROTECTED] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )