[Django] #26971: UnicodeDecodeError with non-ASCII string in quoted URL

[Django]

#26971: UnicodeDecodeError with non-ASCII string in quoted URL
-+-
 Reporter:  Oleg78   |  Owner:  nobody
 Type:  Bug  | Status:  new
Component:  HTTP |Version:  1.8
  handling   |   Keywords:  UnicodeDecodeError UTF-8
 Severity:  Normal   |  windows-1251 URL wsgi
 Triage Stage:   |  Has patch:  0
  Unreviewed |
Easy pickings:  0|  UI/UX:  0
-+-
 Django raises UnicodeDecodeError if there are non UTF-8 characters in the
 url.


 
[https://github.com/django/django/blob/master/django/core/handlers/wsgi.py#L190]:
 {{{
 return path_info.decode(UTF_8)
 }}}

 It doesn't work if the parameter in the URL is not in UTF-8
 {{{/tag/%E7%E0%EA%EB%E0%E4%EA%E0/}}}:
 {{{
 GET /tag/%E7%E0%EA%EB%E0%E4%EA%E0/ => generated 0 bytes in 1 msecs
 (HTTP/1.1 400) 1 headers in 68 bytes (1 switches on core 0)
 Bad Request (UnicodeDecodeError)
 Traceback (most recent call last):
   File "/home/ubuntu/django/lib/python3.4/site-
 packages/django/core/handlers/wsgi.py", line 167, in __call__
 request = self.request_class(environ)
   File "/home/ubuntu/django/lib/python3.4/site-
 packages/django/core/handlers/wsgi.py", line 80, in __init__
 path_info = get_path_info(environ)
   File "/home/ubuntu/django/lib/python3.4/site-
 packages/django/core/handlers/wsgi.py", line 197, in get_path_info
 return path_info.decode(UTF_8)
 UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe7 in position 5:
 invalid continuation byte
 }}}

 With utf url-quoted parameter
 {{{/tag/%D0%B7%D0%B0%D0%BA%D0%BB%D0%B0%D0%B4%D0%BA%D0%B0}}} there is no
 errors, but the old site has used windows-1251 encoding and I need to
 support old links. So I use this dirty hack:
 {{{#!python
 try:
 return path_info.decode(UTF_8)
 except:
 return path_info.decode(windows-1251)
 }}}

 The problem is only in wsgi handler, {{{manage.py runserver}}} handles
 non-utf urls without errors.

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/049.e4df2066e3618ff5298b9cd7362c7731%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #26971: UnicodeDecodeError with non-ASCII string in quoted URL

[Django]

#26971: UnicodeDecodeError with non-ASCII string in quoted URL
-+-
 Reporter:  Oleg78   |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  HTTP handling|  Version:  1.8
 Severity:  Normal   |   Resolution:
 Keywords:  UnicodeDecodeError   | Triage Stage:
  UTF-8 windows-1251 URL wsgi|  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by claudep):

 * needs_better_patch:   => 0
 * needs_tests:   => 0
 * needs_docs:   => 0


Comment:

 This was supposed to be fixed by #19508 (hence the `runserver` not
 failing).

 However, I suspect that in your production deployment, the received URI is
 already percent-decoded higher in the stack (Apache, mod_wsgi,...), so
 Django is receiving `/tag/\xe7\xe0\xea\xeb\xe0\xe4\xea\xe0/` instead of
 `/tag/%E7%E0%EA%EB%E0%E4%EA%E0/`. In that case, we may try to "repercent"
 the URI in case of `UnicodeDecodeError`.

 Loïc, could you advise?

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.16635cbf59557052bb05789354fd7ca8%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #26971: UnicodeDecodeError with non-ASCII string in quoted URL

[Django]

#26971: UnicodeDecodeError with non-ASCII string in quoted URL
-+-
 Reporter:  Oleg78   |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  HTTP handling|  Version:  1.8
 Severity:  Normal   |   Resolution:
 Keywords:  UnicodeDecodeError   | Triage Stage:
  UTF-8 windows-1251 URL wsgi|  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by claudep):

 * cc: loic84 (added)


--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.a08f6d3d922984287ec73415ab6e14eb%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #26971: UnicodeDecodeError with non-ASCII string in quoted URL

[Django]

#26971: UnicodeDecodeError with non-ASCII string in quoted URL
-+-
 Reporter:  Oleg78   |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  HTTP handling|  Version:  1.8
 Severity:  Normal   |   Resolution:
 Keywords:  UnicodeDecodeError   | Triage Stage:  Accepted
  UTF-8 windows-1251 URL wsgi|
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by timgraham):

 * stage:  Unreviewed => Accepted


Comment:

 Not sure about the appropriate resolution, but I could reproduce this
 crash by trying to fetch a URL like `/tag/%E7%E0%EA%EB%E0%E4%EA%E0/` using
 gunicorn as the server.

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.22ae1fafc4d1c3abba2921c742fa96be%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #26971: UnicodeDecodeError with non-ASCII string in quoted URL

[Django]

#26971: UnicodeDecodeError with non-ASCII string in quoted URL
-+-
 Reporter:  Oleg78   |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  HTTP handling|  Version:  1.8
 Severity:  Normal   |   Resolution:
 Keywords:  UnicodeDecodeError   | Triage Stage:  Accepted
  UTF-8 windows-1251 URL wsgi|
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by claudep):

 * has_patch:  0 => 1


Comment:

 Suggested fix in that [https://github.com/django/django/pull/7070 PR].

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.986d65565a198600a19ee4f6b0c7c3c0%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #26971: UnicodeDecodeError with non-ASCII string in quoted URL

[Django]

#26971: UnicodeDecodeError with non-ASCII string in quoted URL
-+-
 Reporter:  Oleg78   |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  HTTP handling|  Version:  1.8
 Severity:  Normal   |   Resolution:
 Keywords:  UnicodeDecodeError   | Triage Stage:  Ready for
  UTF-8 windows-1251 URL wsgi|  checkin
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by loic):

 * stage:  Accepted => Ready for checkin


--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.12c994783ad3bd93f7b0705ad4be3e21%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #26971: UnicodeDecodeError with non-ASCII string in quoted URL

[Django]

#26971: UnicodeDecodeError with non-ASCII string in quoted URL
-+-
 Reporter:  Oleg78   |Owner:  nobody
 Type:  Bug  |   Status:  closed
Component:  HTTP handling|  Version:  1.8
 Severity:  Normal   |   Resolution:  fixed
 Keywords:  UnicodeDecodeError   | Triage Stage:  Ready for
  UTF-8 windows-1251 URL wsgi|  checkin
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Claude Paroz ):

 * status:  new => closed
 * resolution:   => fixed


Comment:

 In [changeset:"48c34f3336cbbc906066636a7aa35270a7b44895" 48c34f3]:
 {{{
 #!CommitTicketReference repository=""
 revision="48c34f3336cbbc906066636a7aa35270a7b44895"
 Fixed #26971 -- Prevented crash with non-UTF-8 incoming PATH_INFO

 Thanks Tim Graham and Loïc Bistuer for the reviews.
 }}}

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.567505f8be370cab478938ba0cf7d2a5%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.