Request data encoding

2007-08-02 Thread Daniel Brandt
I was cleaning up a django app to work with the unicode merge, when I noticed a problem that has taken me a good while to debug, and I still don't have the full picture. I am recieving POST-data that is submitted to my application not via a form or a browser, but from other web applications, acc

Re: Request data encoding

2007-08-02 Thread Jacob Kaplan-Moss
On 8/2/07, Daniel Brandt <[EMAIL PROTECTED]> wrote: > I am recieving POST-data that is submitted to my application not via a > form or a browser, but from other web applications, according to a > known protocol. This data may or may not have the charset of the data > set in the Content-Type header

Re: Request data encoding

2007-08-02 Thread Gábor Farkas
Jacob Kaplan-Moss wrote: > On 8/2/07, Daniel Brandt <[EMAIL PROTECTED]> wrote: >> I am recieving POST-data that is submitted to my application not via a >> form or a browser, but from other web applications, according to a >> known protocol. This data may or may not have the charset of the data >>

Re: Request data encoding

2007-08-02 Thread ludvig.ericson
On Aug 2, 11:02 pm, Gábor Farkas <[EMAIL PROTECTED]> wrote: > Jacob Kaplan-Moss wrote: > > On 8/2/07, Daniel Brandt <[EMAIL PROTECTED]> wrote: > >> I am recieving POST-data that is submitted to my application not via a > >> form or a browser, but from other web applications, according to a > >>

Re: Request data encoding

2007-08-02 Thread Daniel Brandt
On 8/2/07, Jacob Kaplan-Moss <[EMAIL PROTECTED]> wrote: > > On 8/2/07, Daniel Brandt <[EMAIL PROTECTED]> wrote: > > I am recieving POST-data that is submitted to my application not via a > > form or a browser, but from other web applications, according to a > > known protocol. This data may or may

Re: Request data encoding

2007-08-02 Thread Craig Ogg
On 8/2/07, ludvig.ericson <[EMAIL PROTECTED]> wrote: > On Aug 2, 11:02 pm, Gábor Farkas <[EMAIL PROTECTED]> wrote: > > Jacob Kaplan-Moss wrote: > > > On 8/2/07, Daniel Brandt <[EMAIL PROTECTED]> wrote: > > >> I am recieving POST-data that is submitted to my application not via a > > >> form or a b

Re: Request data encoding

2007-08-02 Thread Jacob Kaplan-Moss
On 8/2/07, Craig Ogg <[EMAIL PROTECTED]> wrote: > I am new to django and this list, so I hope this email is constructive > and helpful. Amazingly useful! I'm still digesting all that tasty informational goodness, but I'm pretty sure you're right that we should assume the current default charset

Re: Request data encoding

2007-08-02 Thread Simon Willison
On Aug 2, 9:39 pm, "Jacob Kaplan-Moss" <[EMAIL PROTECTED]> wrote: > Yuck, clients that don't speak HTTP correctly make me angry. > > Reading the RFC, though, I see that since HTTP 1.0 made "charset" > optional, it remains so in HTTP 1.1, and we're supposed to "guess" and > use ISO-8859-1 like you'

Re: Request data encoding

2007-08-02 Thread Jacob Kaplan-Moss
On 8/2/07, Simon Willison <[EMAIL PROTECTED]> wrote: > This is a totally ridiculous flaw with the HTTP spec - you literally > have no reliable way of telling what encoding a request coming in to > your site uses, since you can't be absolutely sure that the user-agent > read a page from your site t

Re: Request data encoding

2007-08-02 Thread Craig Ogg
On 8/2/07, Jacob Kaplan-Moss <[EMAIL PROTECTED]> wrote: > * Try to decode the form data using ``settings.DEFAULT_CHARSET``. In > most cases this'll be UTF-8, but when it's not we can try to assume > that data's being POSTed back in the same encoding we're serving it up > in. > * If that fails and

Re: Request data encoding

2007-08-03 Thread James Tauber
Coincidentally, I mostly recently encountered this problem with form character encoding an hour ago when I tried to use the International Phonetic Alphabet in a comment on Ian Bicking's blog post[1] about how to pronounce Django :-) (note: at time of writing, Ian hadn't approved my comment

Re: Request data encoding

2007-08-03 Thread Daniel Brandt
OK.. I gave it a shot.. not really satisfied with the patch yet, but it's a start. Feel free to suggest improvements (or submit a better, or extended, patch) if need be. Here's the ticket http://code.djangoproject.com/ticket/5076 Hope everyone has a great friday! Regards, Daniel --~--~-

Re: Request data encoding

2007-08-10 Thread Malcolm Tredinnick
On Thu, 2007-08-02 at 15:14 -0700, Craig Ogg wrote: > On 8/2/07, ludvig.ericson <[EMAIL PROTECTED]> wrote: > > On Aug 2, 11:02 pm, Gábor Farkas <[EMAIL PROTECTED]> wrote: > > > Jacob Kaplan-Moss wrote: > > > > On 8/2/07, Daniel Brandt <[EMAIL PROTECTED]> wrote: > > > >> I am recieving POST-data th

Re: Request data encoding

2007-08-10 Thread Malcolm Tredinnick
On Thu, 2007-08-02 at 19:33 -0500, Jacob Kaplan-Moss wrote: > On 8/2/07, Simon Willison <[EMAIL PROTECTED]> wrote: > > This is a totally ridiculous flaw with the HTTP spec - you literally > > have no reliable way of telling what encoding a request coming in to > > your site uses, since you can't b

Re: Request data encoding

2007-08-10 Thread Malcolm Tredinnick
On Sat, 2007-08-11 at 13:16 +1000, Malcolm Tredinnick wrote: [...] > Receiving genuinely bad/invalid data is not uncommon either, as is > obvious as soon as you start running a really anal comment sanitisation > feature or looking at uploads from corporate systems. Trying to silently > change the

Re: Request data encoding

2007-08-11 Thread Daniel Brandt
My only problem with this is that I feel I'm writing framework-code in my view. I'm clearly breaking the separation between application and framework. Of course, no design will ever be perfect and you will always have corner cases like this. Instead of ditching the idea alltogether I think making

Re: Request data encoding

2007-08-11 Thread Malcolm Tredinnick
On Sat, 2007-08-11 at 20:04 +0200, Daniel Brandt wrote: > My only problem with this is that I feel I'm writing framework-code in > my view. I'm clearly breaking the separation between application and > framework. Of course, no design will ever be perfect and you will > always have corner cases lik