[Repoze-dev] [issue100] Configurable character set support for repoze.who and repoze.what
System message: -- status: chatting - deferred __ Repoze Bugs b...@bugs.repoze.org http://bugs.repoze.org/issue100 __ ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org https://lists.repoze.org/mailman/listinfo/repoze-dev
[Repoze-dev] [issue100] Configurable character set support for repoze.who and repoze.what
Tres Seaver tsea...@agendaless.com added the comment: The only correct way to decode values in the payload of an HTTP POST request is to use the charset supplied in the Content-Type header of that request, or fall back to ISO-8859-1. RFC 2616, section 3.7.1, says: When no explicit charset parameter is provided by the sender, media subtypes of the text type are defined to have a default charset value of ISO-8859-1 when received via HTTP. -- status: resolved - chatting __ Repoze Bugs b...@bugs.repoze.org http://bugs.repoze.org/issue100 __ ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev
[Repoze-dev] [issue100] Configurable character set support for repoze.who and repoze.what
Yuen Ho Wong wyue...@gmail.com added the comment: I'm not entirely sure why this is not actionable on any realistic level, and what you mean by lower layers. Perhaps you can explain further or point me to a previous discussion if this has been talked about before? -- status: resolved - chatting __ Repoze Bugs b...@bugs.repoze.org http://bugs.repoze.org/issue100 __ ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev
[Repoze-dev] [issue100] Configurable character set support for repoze.who and repoze.what
Chris McDonough chr...@plope.com added the comment: I mean that I personally don't know how to change the software based on your suggestions so far in a way that adds significant value, and that if WSGI2 will handle the encoding and decoding issues, maybe this should just do something better with r.who then. OTOH, I'd encourage a set of patches against the WSGI 1 version that implement what you believe would help. In any case, let's at least please stop talking via the bugtracker about this issue; it would be better to take it to the repoze-dev maillist or the #repoze IRC channel on irc.freenode.net (I am mcdonc there). -- status: chatting - resolved __ Repoze Bugs b...@bugs.repoze.org http://bugs.repoze.org/issue100 __ ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev
[Repoze-dev] [issue100] Configurable character set support for repoze.who and repoze.what
Chris McDonough chr...@plope.com added the comment: I believe this. On the other hand, I'm not sure that a 'charset' setting dedicated to repoze.who or to repoze.what is really sufficient. Who will set the charset? What will it be set to? What does it mean to individual plugins? What would they need to do to comply with the charset?What does holistically support mean? This is a bit of a minefield. There is a fundamental lack of support at very basic levels for charset support. Even HTML form posts don't send along the charset of the posted data: it's presumed that the POST data will be in the charset of the HTML page which contained the rendered form. If your form handler doesn't know what that charset is, you're out of luck. So I'm not sure WSGI can really do much here in general. In short: what would you change, concretely to make this any better? -- status: unread - chatting __ Repoze Bugs b...@bugs.repoze.org http://bugs.repoze.org/issue100 __ ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev
[Repoze-dev] [issue100] Configurable character set support for repoze.who and repoze.what
Yuen Ho Wong wyue...@gmail.com added the comment: Well I think the WSGI 1.x spec has made a mistake of mandating all strings in environ to be byte strings while not defining a global environment variable to give middlewares a hint of how to decode the byte strings. This is a recognized problem that is address in WSGI 2 by mandating strings to be unicode. The problem with not knowing how to decode byte strings is not not knowing how to decode in the handlers but how to decode in the middlewares, which is supposed to be application agnostic. To limit this problem to just repoze.who, say I have an IIdentifier that wants to remember credentials according to different charsets on a per request basis. In a perfect world, the server will set a well known variable in the request on the first opportunity, and the plugin will just look for it and encode accordingly. But in WSGI, there's no request object, there's only an environ, so we are stuck with that. So in this less perfect world, there would be a well-known charset variable in the environ to give hints to middlewares and the applications. But there isn't, so we application developers have to invent one. Right now, every framework deals with it differently, but at the end of the day, there is a threadlocal charset variable that __handlers__ can use. There is no equivalent in repoze.who and repoze.what. As I have already said, until Py3k takes off and we are all using WSGI 2, this will be a problem we are stuck with and middlewares will need to deal with it. I have already proposed 3 solutions in comment #1. I'm in favor of solution number 2. To answer your questions. Yes, just having a charset for repoze.who will not solve all the problems of decoding in WSGI apps, but at least it's half of the solution. I believe the repoze.who middleware should take a parameter in the constructor, such that it can set it into the environ as early as possible. To plugins, this serves as a hint - meaning only a default - charset to decode bytestrings. There's no compliance forced on plugins, but it only serves as a very helpful clue. Holistic support simply means 1) have a globally visible charset variable for all of repoze.who at any scope, 2) all the plugins will make a best effort to decode according to the global charset. I hope this clears up the issue. __ Repoze Bugs b...@bugs.repoze.org http://bugs.repoze.org/issue100 __ ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev
[Repoze-dev] [issue100] Configurable character set support for repoze.who and repoze.what
New submission from Yuen Ho Wong wyue...@gmail.com: It seems that from the design to the implementation of repoze.who and repoze.what, including its plugins, have completely failed to support holistically any charset other then Latin-1. As of now, there is not an option global to repoze.who, and subsequently repoze.what, to indicate to the plugins as to what charset to decode the HTTP params to. This can cause serious problems for websites written in exotic charsets like Big-5. Even for websites written in UTF-8, there is still not an option to guarantee the plugins the correct charset codec. I was looking at the source code for repoze.who, AuthTktCookie, *Forms, and repoze.who.plugins.sa, and it seems that putting a `charset` variable into the `environ` is the best way to go for reasons of global visibility (makes me want to lobby the WSGI people to put this in WSGI 1.1 and 2). The problem with this is I'm not sure if there's any frameworks/servers out there that already do this. Another way to do it is to name the variable `repoze.who.charset`. Yet another was is to have plugins look for a key `charset` from the identity, which localize this variable's visibility to just repoze.who and repoze.what plugins. -- assignedto: chrism messages: 273 nosy: chrism, wyuenho priority: bug status: unread title: Configurable character set support for repoze.who and repoze.what __ Repoze Bugs b...@bugs.repoze.org http://bugs.repoze.org/issue100 __ ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev
