Re: [Django] #24500: Django runtests - 3 tests fail on windows due to encoding troubles

[Django]

#24500: Django runtests - 3 tests fail on windows due to encoding troubles
-+-
 Reporter:  pakal|Owner:  nobody
 Type:  Bug  |   Status:  closed
Component:   |  Version:  1.8rc1
  Internationalization   |
 Severity:  Release blocker  |   Resolution:  fixed
 Keywords:   | Triage Stage:  Ready for
 |  checkin
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by claudep):

 * status:  new => closed
 * resolution:   => fixed


--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/063.f023dc3b2e54afb08f66a16007c2fd02%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #24500: Django runtests - 3 tests fail on windows due to encoding troubles

[Django]

#24500: Django runtests - 3 tests fail on windows due to encoding troubles
-+-
 Reporter:  pakal|Owner:  nobody
 Type:  Bug  |   Status:  new
Component:   |  Version:  1.8rc1
  Internationalization   |
 Severity:  Release blocker  |   Resolution:
 Keywords:   | Triage Stage:  Ready for
 |  checkin
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Claude Paroz ):

 In [changeset:"72de42cd401613197902370c812eaf607244d87c" 72de42cd]:
 {{{
 #!CommitTicketReference repository=""
 revision="72de42cd401613197902370c812eaf607244d87c"
 [1.8.x] Refs #24500 -- Avoided locale.getpreferredencoding in makemessages

 Fixes a regression introduced in 53c2cf1e.
 Backport of 3f4e77840 from master.
 }}}

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/063.d559f67f4eac2f5689800b1d763705be%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #24500: Django runtests - 3 tests fail on windows due to encoding troubles

[Django]

#24500: Django runtests - 3 tests fail on windows due to encoding troubles
-+-
 Reporter:  pakal|Owner:  nobody
 Type:  Bug  |   Status:  new
Component:   |  Version:  1.8rc1
  Internationalization   |
 Severity:  Release blocker  |   Resolution:
 Keywords:   | Triage Stage:  Ready for
 |  checkin
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Claude Paroz ):

 In [changeset:"3f4e77840249c533b08ef9f1d03d7e3387ed2b4c" 3f4e778]:
 {{{
 #!CommitTicketReference repository=""
 revision="3f4e77840249c533b08ef9f1d03d7e3387ed2b4c"
 Refs #24500 -- Avoided locale.getpreferredencoding in makemessages

 Fixes a regression introduced in 53c2cf1e.
 }}}

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/063.3d25b6faef91be45f18b171d8303e772%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #24500: Django runtests - 3 tests fail on windows due to encoding troubles

[Django]

#24500: Django runtests - 3 tests fail on windows due to encoding troubles
-+-
 Reporter:  pakal|Owner:  nobody
 Type:  Bug  |   Status:  new
Component:   |  Version:  1.8rc1
  Internationalization   |
 Severity:  Release blocker  |   Resolution:
 Keywords:   | Triage Stage:  Ready for
 |  checkin
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by claudep):

 You didn't have a problem before because you have an English locale, and
 the gettext response doesn't include any non-ascii character for you.

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/063.58a33377201b10aab3abc6a1be82f0e2%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #24500: Django runtests - 3 tests fail on windows due to encoding troubles

[Django]

#24500: Django runtests - 3 tests fail on windows due to encoding troubles
-+-
 Reporter:  pakal|Owner:  nobody
 Type:  Bug  |   Status:  new
Component:   |  Version:  1.8rc1
  Internationalization   |
 Severity:  Release blocker  |   Resolution:
 Keywords:   | Triage Stage:  Ready for
 |  checkin
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by timgraham):

 * stage:  Accepted => Ready for checkin


Comment:

 Patch works on my Windows. On the other hand, I didn't have a problem
 before this ticket.

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/063.83778fe676b48097ddf32028bb88f57e%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #24500: Django runtests - 3 tests fail on windows due to encoding troubles

[Django]

#24500: Django runtests - 3 tests fail on windows due to encoding troubles
--+
 Reporter:  pakal |Owner:  nobody
 Type:  Bug   |   Status:  new
Component:  Internationalization  |  Version:  1.8rc1
 Severity:  Release blocker   |   Resolution:
 Keywords:| Triage Stage:  Accepted
Has patch:  1 |  Needs documentation:  0
  Needs tests:  0 |  Patch needs improvement:  0
Easy pickings:  0 |UI/UX:  0
--+
Changes (by claudep):

 * has_patch:  0 => 1


Comment:

 PR https://github.com/django/django/pull/4425
 `locale.getpreferredencoding(False)` is not usable. For example, it
 returns `ANSI_X3.4-1968` on my system!
 The patch needs to be tested on Windows now.

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/063.02424cc573c4bc6b23ac3885877ffe38%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #24500: Django runtests - 3 tests fail on windows due to encoding troubles

[Django]

#24500: Django runtests - 3 tests fail on windows due to encoding troubles
--+
 Reporter:  pakal |Owner:  nobody
 Type:  Bug   |   Status:  new
Component:  Internationalization  |  Version:  1.8rc1
 Severity:  Release blocker   |   Resolution:
 Keywords:| Triage Stage:  Accepted
Has patch:  0 |  Needs documentation:  0
  Needs tests:  0 |  Patch needs improvement:  0
Easy pickings:  0 |UI/UX:  0
--+
Changes (by claudep):

 * status:  closed => new
 * severity:  Normal => Release blocker
 * version:  master => 1.8rc1
 * has_patch:  1 => 0
 * resolution:  fixed =>
 * stage:  Ready for checkin => Accepted


Comment:

 Now those tests are failing on my system :-/ I'll try to debug this issue.

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/063.03a1e6f57aed8b604324977128d7ba7a%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #24500: Django runtests - 3 tests fail on windows due to encoding troubles

[Django]

#24500: Django runtests - 3 tests fail on windows due to encoding troubles
-+-
 Reporter:  pakal|Owner:  nobody
 Type:  Bug  |   Status:  closed
Component:   |  Version:  master
  Internationalization   |
 Severity:  Normal   |   Resolution:  fixed
 Keywords:   | Triage Stage:  Ready for
 |  checkin
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Tim Graham ):

 In [changeset:"dc2cff5fcbbff207dc99d262e9f13de954c7267e" dc2cff5]:
 {{{
 #!CommitTicketReference repository=""
 revision="dc2cff5fcbbff207dc99d262e9f13de954c7267e"
 [1.8.x] Fixed #24500 -- Fixed makemessages encoding problems retrieving
 gettext version.

 Backport of 53c2cf1e7b3f1c9a7794fcc5c4669145f351b2fa from master
 }}}

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/063.4b7304f8958ddaa52ae551022e378229%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #24500: Django runtests - 3 tests fail on windows due to encoding troubles

[Django]

#24500: Django runtests - 3 tests fail on windows due to encoding troubles
-+-
 Reporter:  pakal|Owner:  nobody
 Type:  Bug  |   Status:  closed
Component:   |  Version:  master
  Internationalization   |
 Severity:  Normal   |   Resolution:  fixed
 Keywords:   | Triage Stage:  Ready for
 |  checkin
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Tim Graham ):

 * status:  new => closed
 * resolution:   => fixed


Comment:

 In [changeset:"53c2cf1e7b3f1c9a7794fcc5c4669145f351b2fa" 53c2cf1e]:
 {{{
 #!CommitTicketReference repository=""
 revision="53c2cf1e7b3f1c9a7794fcc5c4669145f351b2fa"
 Fixed #24500 -- Fixed makemessages encoding problems retrieving gettext
 version.
 }}}

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/063.07a7ad7c4c49a47a4217b1ffc2adc261%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #24500: Django runtests - 3 tests fail on windows due to encoding troubles

[Django]

#24500: Django runtests - 3 tests fail on windows due to encoding troubles
-+-
 Reporter:  pakal|Owner:  nobody
 Type:  Bug  |   Status:  new
Component:   |  Version:  master
  Internationalization   |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Ready for
 |  checkin
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by claudep):

 Apart from a minor note on the PR, it's fine for me.

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/063.0a581bda59eeda5c318fa8574f3c5265%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #24500: Django runtests - 3 tests fail on windows due to encoding troubles

[Django]

#24500: Django runtests - 3 tests fail on windows due to encoding troubles
-+-
 Reporter:  pakal|Owner:  nobody
 Type:  Bug  |   Status:  new
Component:   |  Version:  master
  Internationalization   |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Ready for
 |  checkin
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by timgraham):

 * stage:  Accepted => Ready for checkin


Comment:

 [https://github.com/django/django/pull/4359 PR]; tests seem fine on CI and
 my Windows setup. Claude does the code look fine?

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/063.7b9d249d45d17991865e402a2bb59ab6%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #24500: Django runtests - 3 tests fail on windows due to encoding troubles

[Django]

#24500: Django runtests - 3 tests fail on windows due to encoding troubles
--+
 Reporter:  pakal |Owner:  nobody
 Type:  Bug   |   Status:  new
Component:  Internationalization  |  Version:  master
 Severity:  Normal|   Resolution:
 Keywords:| Triage Stage:  Accepted
Has patch:  1 |  Needs documentation:  0
  Needs tests:  0 |  Patch needs improvement:  0
Easy pickings:  0 |UI/UX:  0
--+

Comment (by timgraham):

 Are you able to create a GitHub pull request?

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/063.5623e3a6240f3345702931f72945a7e8%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #24500: Django runtests - 3 tests fail on windows due to encoding troubles

[Django]

#24500: Django runtests - 3 tests fail on windows due to encoding troubles
--+
 Reporter:  pakal |Owner:  nobody
 Type:  Bug   |   Status:  new
Component:  Internationalization  |  Version:  master
 Severity:  Normal|   Resolution:
 Keywords:| Triage Stage:  Accepted
Has patch:  1 |  Needs documentation:  0
  Needs tests:  0 |  Patch needs improvement:  0
Easy pickings:  0 |UI/UX:  0
--+
Changes (by claudep):

 * has_patch:  0 => 1


--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/063.367a612170d9ccc004c294fbe09cc0f4%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #24500: Django runtests - 3 tests fail on windows due to encoding troubles

[Django]

#24500: Django runtests - 3 tests fail on windows due to encoding troubles
--+
 Reporter:  pakal |Owner:  nobody
 Type:  Bug   |   Status:  new
Component:  Internationalization  |  Version:  master
 Severity:  Normal|   Resolution:
 Keywords:| Triage Stage:  Accepted
Has patch:  0 |  Needs documentation:  0
  Needs tests:  0 |  Patch needs improvement:  0
Easy pickings:  0 |UI/UX:  0
--+

Comment (by pakal):

 Yes I tried it, but it broke other assumptions (especially tests dedicated
 to verify real file encoding troubles).

 Furthermore, I'm afraid it might lead to new cases of mojibake, since an
 ANSI (ex. latin1) string might, by error, be treated as an utf8 string
 (and thus unexpected unicode characters be formed by grouping together 2
 our more single-byte characters). Most of the time we get "invalid
 continuation byte" in such cases, but if one has bad luck

 I guess our best chance is to bet on empirical evidence, and assume that
 xgettext tools '''always''' return utf-8 strings, '''unless''' when called
 for side tasks like displaying version info. As long as tests pass on both
 *nix and windows, python2 and python3, on non-english computers, i'm
 rather confident about the robustness of this solution (test coverage is
 pretty strong on these i18n encoding troubles).

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/063.7cf5b2f78f408f7b075102664c47622e%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #24500: Django runtests - 3 tests fail on windows due to encoding troubles

[Django]

#24500: Django runtests - 3 tests fail on windows due to encoding troubles
--+
 Reporter:  pakal |Owner:  nobody
 Type:  Bug   |   Status:  new
Component:  Internationalization  |  Version:  master
 Severity:  Normal|   Resolution:
 Keywords:| Triage Stage:  Accepted
Has patch:  0 |  Needs documentation:  0
  Needs tests:  0 |  Patch needs improvement:  0
Easy pickings:  0 |UI/UX:  0
--+

Comment (by claudep):

 What about the try/except approach, something like:
 {{{
 try:
 stdout = stdout.decode('utf-8')
 except UnicodeDecodeError:
 stdout = stdout.decode(preferred_encoding)
 }}}

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/063.2b9d0bdc4dde15937a1bc51cdbd91a4f%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #24500: Django runtests - 3 tests fail on windows due to encoding troubles

[Django]

#24500: Django runtests - 3 tests fail on windows due to encoding troubles
--+
 Reporter:  pakal |Owner:  nobody
 Type:  Bug   |   Status:  new
Component:  Internationalization  |  Version:  master
 Severity:  Normal|   Resolution:
 Keywords:| Triage Stage:  Accepted
Has patch:  0 |  Needs documentation:  0
  Needs tests:  0 |  Patch needs improvement:  0
Easy pickings:  0 |UI/UX:  0
--+

Comment (by pakal):

 Hum I tried different ways of catching decoding exceptions to provide a
 fallback, but it broke more tests elsewhere

 In the end, I just special-cased the "xgettest -V" call, which doesn't
 respect utf-8 encoding, and let the rest encode/decode as is (CF attached
 patch). Tested on windows 7, on latest python2 and python3 interpreters.

 In the patch I also added some encoding details to an error message (to
 help, since encoding troubles are recurring), but maybe it's too much
 information, I don't know.

 I'm amazed, though, that xgettext, which is supposed to deal with
 internationalization, doesn't provide parameters to controls its output
 encoding, doesn't say anything about it in its docs (afaik), and has
 different stdout encodings in its different commands...

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/063.9c84cbdf7c4730cd00a9b3ac07a73587%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #24500: Django runtests - 3 tests fail on windows due to encoding troubles

[Django]

#24500: Django runtests - 3 tests fail on windows due to encoding troubles
--+
 Reporter:  pakal |Owner:  nobody
 Type:  Bug   |   Status:  new
Component:  Internationalization  |  Version:  master
 Severity:  Normal|   Resolution:
 Keywords:| Triage Stage:  Accepted
Has patch:  0 |  Needs documentation:  0
  Needs tests:  0 |  Patch needs improvement:  0
Easy pickings:  0 |UI/UX:  0
--+
Changes (by pakal):

 * Attachment "0001-Fix-gettext-tools-output-encoding-troubles.patch"
 added.


--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/063.167bafffb6ef1577d05c8a18650729ec%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #24500: Django runtests - 3 tests fail on windows due to encoding troubles

[Django]

#24500: Django runtests - 3 tests fail on windows due to encoding troubles
--+
 Reporter:  pakal |Owner:  nobody
 Type:  Bug   |   Status:  new
Component:  Internationalization  |  Version:  master
 Severity:  Normal|   Resolution:
 Keywords:| Triage Stage:  Accepted
Has patch:  0 |  Needs documentation:  0
  Needs tests:  0 |  Patch needs improvement:  0
Easy pickings:  0 |UI/UX:  0
--+

Comment (by claudep):

 Yes, let's fix this at `gettext_popen_wrapper` level for now.

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/063.ae9d5d72e581ed7040c165e03937d6bd%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #24500: Django runtests - 3 tests fail on windows due to encoding troubles

[Django]

#24500: Django runtests - 3 tests fail on windows due to encoding troubles
--+
 Reporter:  pakal |Owner:  nobody
 Type:  Bug   |   Status:  new
Component:  Internationalization  |  Version:  master
 Severity:  Normal|   Resolution:
 Keywords:| Triage Stage:  Accepted
Has patch:  0 |  Needs documentation:  0
  Needs tests:  0 |  Patch needs improvement:  0
Easy pickings:  0 |UI/UX:  0
--+
Changes (by timgraham):

 * cc: claudep (added)
 * version:  1.7 => master
 * stage:  Unreviewed => Accepted


Comment:

 Claude, could you advise on the best implementation?

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/063.260b7c20a6a99bd0047982fbb65ea73b%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #24500: Django runtests - 3 tests fail on windows due to encoding troubles

[Django]

#24500: Django runtests - 3 tests fail on windows due to encoding troubles
-+-
 Reporter:  pakal|Owner:  nobody
 Type:  Bug  |   Status:  new
Component:   |  Version:  1.7
  Internationalization   |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by pakal):

 Thanks

 I guess that's why -> your "xgettext -V" output is all-english, so ascii-
 compatible, hence that it be treated as utf8, a windows codepage, or
 ascii, is the same for django, and no error occurs. But for non-ascii
 outputs like my french "xgettext -V", decoding breaks because the output
 of Popen is cp1252 with accentuated letters, not utf8.

 So we should replace "stdout = stdout.decode('utf-8')" by a proper
 operation with DEFAULT_LOCALE_ENCODING. I don't know what's best -
 normalize everything at popen_wrapper level (but it might break lots of
 code that rely on raw DEFAULT_LOCALE_ENCODING bytes), or handle stuffs
 properly in the new gettext_popen_wrapper() ? The latter i guess ?

 Note that django1.7 didn't have these things, so no tests broke.

 When was know the way to go, I can draft a patch to solve that non-ascii-
 encoding buglet.

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/063.7547a84b464d0b529718e8313f0c0c36%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #24500: Django runtests - 3 tests fail on windows due to encoding troubles

[Django]

#24500: Django runtests - 3 tests fail on windows due to encoding troubles
-+-
 Reporter:  pakal|Owner:  nobody
 Type:  Bug  |   Status:  new
Component:   |  Version:  1.7
  Internationalization   |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by timgraham):

 {{{
 >>> locale.getdefaultlocale()
 ('en_US', 'cp1252')

 $ xgettext -V
 xgettext.exe (GNU gettext-tools) 0.17
 Copyright (C) 1995-1998, 2000-2007 Free Software Foundation, Inc.
 License GPLv3+: GNU GPL version 3 or later
 
 This is free software: you are free to change and redistribute it.
 There is NO WARRANTY, to the extent permitted by law.
 Written by Ulrich Drepper.
 }}}

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/063.05562030af6079b0cb8f8f07365fd352%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #24500: Django runtests - 3 tests fail on windows due to encoding troubles

[Django]

#24500: Django runtests - 3 tests fail on windows due to encoding troubles
-+-
 Reporter:  pakal|Owner:  nobody
 Type:  Bug  |   Status:  new
Component:   |  Version:  1.7
  Internationalization   |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by pakal):

 Weird, I updated to python 2.7.9 and tried with cmd.exe or git bash, still
 the same problem.

 I continued investigating, problems occur when django tries to lookup
 xgettext version.

 In gettext_popen_wrapper(), the output of
 django.core.management.utils.popen_wrapper is expected by django to be
 utf8 bytes, on python2 :

 {{{
 if six.PY2:
 stdout = stdout.decode('utf-8')
 }}}

 However, from what I see in the initial popen_wrapper(), there are no
 reasons for stdout to be unicode on python2, only stderr gets converted,
 and Popen (AFAIK) outputs bytes (in cp1252 encoding, in that case).

 {{{
 def  popen_wrapper(args, os_err_exc_type=CommandError):
 """
 Friendly wrapper around Popen.

 Returns stdout output, stderr output and OS status code.
 """
 print "USING ENCODING >", DEFAULT_LOCALE_ENCODING   # outputs
 "cp1252"
 try:
 p = Popen(args, shell=False, stdout=PIPE, stderr=PIPE,
 close_fds=os.name != 'nt', universal_newlines=True)
 except OSError as e:
 strerror = force_text(e.strerror, DEFAULT_LOCALE_ENCODING,
   strings_only=True)
 six.reraise(os_err_exc_type, os_err_exc_type('Error executing %s:
 %s' %
 (args[0], strerror)), sys.exc_info()[2])
 output, errors = p.communicate()
 return (
 output,
 force_text(errors, DEFAULT_LOCALE_ENCODING, strings_only=True),
 p.returncode
 )
 }}}

 This is a mystery to me... could anybody check the intermediate values of
 this "xgettext -V" stdout, as well as the system encodings
 (locale.getdefaultlocale() and stuffs), on a machine which doesn't fail
 these tests ?

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/063.a62c670cfee4adba0fab2638994f2b0c%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #24500: Django runtests - 3 tests fail on windows due to encoding troubles

[Django]

#24500: Django runtests - 3 tests fail on windows due to encoding troubles
-+-
 Reporter:  pakal|Owner:  nobody
 Type:  Bug  |   Status:  new
Component:   |  Version:  1.7
  Internationalization   |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by timgraham):

 They pass for me on Python 2.7.9 or Python 3.4.2 on Windows Vista 32-bit,
 xgettext 0.17, Git bash shell.

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/063.46a759a5b28d3ed08a33e65f71720fa0%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #24500: Django runtests - 3 tests fail on windows due to encoding troubles

[Django]

#24500: Django runtests - 3 tests fail on windows due to encoding troubles
-+-
 Reporter:  pakal|Owner:  nobody
 Type:  Bug  |   Status:  new
Component:   |  Version:  1.7
  Internationalization   |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by pakal):

 * needs_better_patch:   => 0
 * needs_tests:   => 0
 * needs_docs:   => 0


Old description:

> On a my python2.7.3 install (32 bits), a today's checkout of django's
> master doesn't pass all tests - three i18n tests fails due to
> UnicodeDecodeError.
>
> I'm aware it might be due to locales
> (https://docs.djangoproject.com/en/dev/internals/contributing/writing-
> code/unit-tests/#many-test-failures-with-unicodeencodeerror), but windows
> doesn't have such packages that I'm aware of.
>
> My win32 "xgettext --version" is  "xgettext (GNU gettext-tools) 0.17"
>
> I've followed the bug down to the xgettext invocation, where it seems the
> test expects utf8 output, whereas my xgettext-win32 version of course
> outputs ANSI (french locale) text.
>
> {{{
> UnicodeDecodeError('utf8', "xgettext (GNU gettext-tools) 0.17\nCopyright
> (C) 1995-1998, 2000-2007 Free Software Foundation, Inc.\nLicence GPLv3+ :
> GNU GPL version 3 ou ult\xe9rieure
> \nCeci est un logiciel libre : vous
> pouvez le modifier et le redistribuer.\nIl n'y a PAS DE GARANTIE, dans la
> mesure de ce que permet la loi.\n\xc9crit par Ulrich
> Drepper.\n", 141, 142, 'invalid continuation byte')
> }}}
>
> Are these tests supposed to pass on windows as well as on *nix systems ?
> Are there specific requirements regarding xgettext or third-party
> packages ?
>
> {{{
>
> examining files with the extensions: .js
> ignoring file code.sample in .
> ignoring file not_utf8.sample in .
> ignoring file __init__.py in .
> ignoring file ignored.html in .\ignore_dir
> ignoring file media_ignored.html in .\media_root
> ignoring file static_ignored.html in .\static
> ignoring file comments.thtml in .\templates
> ignoring file empty.html in .\templates
> ignoring file plural.djtpl in .\templates
> ignoring file template_with_error.tpl in .\templates
> ignoring file test.html in .\templates
> ignoring file xxx_ignored.html in .\templates
> ignoring file ignored.html in .\templates\subdir
> processing file javascript.js in .
> UnicodeDecodeError: skipped file javascript.js in .
> processing file javascript.js in .\someapp\static
> UnicodeDecodeError: skipped file javascript.js in .\someapp\static
> processing file javascript_ignored.js in .\static
> UnicodeDecodeError: skipped file javascript_ignored.js in .\static
> processing locale de
>

> ==
> FAIL: test_default_root_settings
> (i18n.test_extraction.JavascriptExtractorTests)
> --
> Traceback (most recent call last):
>   File "P:\Websites\django\django\test\utils.py", line 180, in inner
> return test_func(*args, **kwargs)
>   File "P:\Websites\django\tests\i18n\test_extraction.py", line 454, in
> test_default_root_settings
> _, po_contents = self._run_makemessages(domain='djangojs')
>   File "P:\Websites\django\tests\i18n\test_extraction.py", line 67, in
> _run_makemessages
> self.assertTrue(os.path.exists(self.PO_FILE))
> AssertionError: False is not true
>
> ==
> FAIL: test_javascript_literals
> (i18n.test_extraction.JavascriptExtractorTests)
> --
> Traceback (most recent call last):
>   File "P:\Websites\django\tests\i18n\test_extraction.py", line 422, in
> test_javascript_literals
> _, po_contents = self._run_makemessages(domain='djangojs')
>   File "P:\Websites\django\tests\i18n\test_extraction.py", line 67, in
> _run_makemessages
> self.assertTrue(os.path.exists(self.PO_FILE))
> AssertionError: False is not true
>
> ==
> FAIL: test_media_static_dirs_ignored
> (i18n.test_extraction.JavascriptExtractorTests)
> --
> Traceback (most recent call last):
>   File "P:\Websites\django\django\test\utils.py", line 180, in inner
> return test_func(*args, **kwargs)
>   File