Re: Enforce the use of a unicode string in settings.LANGUAGES

[Łukasz Rekucki]

Hi everyone,

First I'd like to say I got bitten by this in the past. What worries
me the most in the original report is the TEMPLATE_DEBUG part. IMHO,
this should fail loudly regardless of any debug settings.

As for other stuff:

On 22 January 2014 18:34, Ramiro Morales  wrote:
> I think you are onto something re: the fact that we don't make clear
> that our ugettext*() functions fail to accept encoded literals with
> characters outside ASCII under Python 2.x. even when the encoding
> metadata is correct.

I think everyone is forgetting  that those are *u*gettext() functions
and they work fine with any literals as long as the argument type is
unicode, because that is the only type they know how to handle. The
implicit bytes->text conversion in Python 2 makes this a little less
obvious, but the expected argument type is in the name.

IMHO, It would be better for everyone if ugettext_lazy() and friends
fail immediately when given anything other then text (unicode on
Python 2, str on Python 3), but it's probably too late for that now.
At least until Python 2 support is dropped completely.


-- 
Łukasz Rekucki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAEZs-E%2BVZ-Hnnm_%3D3Mx9XGS-xX9AfZZMEkAbWukhJf9yk0nvmw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Enforce the use of a unicode string in settings.LANGUAGES

[Ramiro Morales]

On Wed, Jan 22, 2014 at 1:12 PM, Henrique Romano  wrote:
>
> That was just an example, ugettext_* returns a functional.proxy, which
> doesn't try to render the string when you print it on the screen, but if you
> try to do it you will see the same result.

You are right.

On Wed, Jan 22, 2014 at 1:58 PM, Henrique Romano  wrote:
>
> On Wed, Jan 22, 2014 at 2:38 PM, Shai Berger  wrote:
>> You might want to take this up with django-cms.
>
> Not exactly true.  Even though my problem was with django-cms, if you grep 
> django source code for "LANGUAGES"  you will find this templatetag:
>
> # templatetags/i18n.py
>  15 class GetAvailableLanguagesNode(Node):
>  16 def __init__(self, variable):
>  17 self.variable = variable
>  18
>  19 def render(self, context):
>  20 from django.conf import settings
>  21 context[self.variable] = [(k, translation.ugettext(v)) for k, v 
> in settings.LANGUAGES]
>  22 return ''
>
>  It is also translating the given language name.

Yes, I think that what both pieces of code are doing there is assume
the LANGUAGES setting in effect is the one shipped with Django where:

- All the original language names are in English (covered by ASCII)
- Most of them are translated by the Django translators and shipped
with Django 
(https://github.com/django/django/blob/master/django/conf/locale/en/LC_MESSAGES/django.po#L16)
so translations are readily available.

So it's relying on the fact there there is no need to go outside ASCII
when working with original language names.

I think you are onto something re: the fact that we don't make clear
that our ugettext*() functions fail to accept encoded literals with
characters outside ASCII under Python 2.x. even when the encoding
metadata is correct.

My theory is that the facts that documentation is a bit unclear about
the requirements these source strings arguments should comply with (or
rather, what are the data types supported by these functions) and that
using another language than English as translate-from language doesn't
seem like a common setup have contributed to this not getting too much
visibility until now.

In Python 3 (3.3) this problem doesn seem to exist. See tests below.

> My suggestion is to update the documentation saying that the developer should 
> use ugettext_lazy for the language name if they want it translated

We have something along these lines. See (last two bullets)
https://docs.djangoproject.com/en/1.6/topics/i18n/translation/#how-django-discovers-language-preference

> *or* at least use a unicode string.

I will try to get some feedback from some more experienced devs to if
there is anything to be done about this in code and/or docs.


$ cat a.py
#  -*- coding: utf-8 -*-

import sys
from django.utils.translation import ugettext, ugettext_lazy, ugettext_noop

print("Python version: %s" % sys.version)

a = ugettext_noop("Portugués")

print(a)
print(type(a))
b = u'%s' % a # try to get the lazy placeholder to evaluate itself.
print(b)
print(type(b))

$ file a.py
a.py: Python script, UTF-8 Unicode text executable

PYTHONPATH=.:~/django/upstream
DJANGO_SETTINGS_MODULE=lang_i18n.settings python a.py
Python version: 2.7.3 (default, Sep 26 2013, 20:03:06)
[GCC 4.6.3]
Portugués

Traceback (most recent call last):
  File "a.py", line 12, in 
b = u'%s' % a # try to get the lazy placeholder to evaluate isself.
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position
7: ordinal not in range(128)


$ PYTHONPATH=.:~/django/upstream
DJANGO_SETTINGS_MODULE=lang_i18n.settings python3.3 a.py
Python version: 3.3.3 (default, Dec 27 2013, 19:27:19)
[GCC 4.6.3]
Portugués

Portugués



Thanks!

-- 
Ramiro Morales
@ramiromorales

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAO7PdF_9%3Dxp0SCkwzmM-rRDTE7X4Yn_UhDaFUuecKSw29jXXZQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Enforce the use of a unicode string in settings.LANGUAGES

[Shai Berger]

On Wednesday 22 January 2014 18:58:26 Henrique Romano wrote:
> On Wed, Jan 22, 2014 at 2:38 PM, Shai Berger  wrote:
> > I don't think Django should take responsibility for a 3rd-party package
> > which
> > decides that some part of a setting should be translatable whether the
> > user said so or not.
> > 
> > You might want to take this up with django-cms.
> 
> Not exactly true.  Even though my problem was with django-cms, if you grep
> django source code for "LANGUAGES"  you will find this templatetag:
> 
> # templatetags/i18n.py
>  15 class GetAvailableLanguagesNode(Node):
>  16 def __init__(self, variable):
>  17 self.variable = variable
>  18
>  19 def render(self, context):
>  20 from django.conf import settings
>  21 context[self.variable] = [(k, translation.ugettext(v)) for k, v
> in settings.LANGUAGES]
>  22 return ''
> 
>  It is also translating the given language name.

You are right. Further, the documentation[1] specifically says that language 
names get translated, whether or not you use this tag.

>  My suggestion is to
> update the documentation saying that the developer should use ugettext_lazy
> for the language name if they want it translated *or* at least use a
> unicode string.

That's not accurate -- the documentation should say that language names get 
translated whether or not you use ugettext_lazy, if you use RequestContext or 
the i18n tags, and *recommend* that anything you put there be either 
translated with ugettext_lazy or be a unicode ubject. It is still valid to use 
a non-unicode string if:

1) The value is all ASCII
2) The value has translations in all used languages.

(you saw the error because your language name does not have a translation, and 
thus was used as given instead).

Bugs and patches welcome,

Shai.

[1] https://docs.djangoproject.com/en/dev/topics/i18n/translation/#other-tags

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/201401221915.45177.shai%40platonix.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Enforce the use of a unicode string in settings.LANGUAGES

[Henrique Romano]

On Wed, Jan 22, 2014 at 2:38 PM, Shai Berger  wrote:

> I don't think Django should take responsibility for a 3rd-party package
> which
> decides that some part of a setting should be translatable whether the user
> said so or not.
>
> You might want to take this up with django-cms.


Not exactly true.  Even though my problem was with django-cms, if you grep
django source code for "LANGUAGES"  you will find this templatetag:

# templatetags/i18n.py
 15 class GetAvailableLanguagesNode(Node):
 16 def __init__(self, variable):
 17 self.variable = variable
 18
 19 def render(self, context):
 20 from django.conf import settings
 21 context[self.variable] = [(k, translation.ugettext(v)) for k, v
in settings.LANGUAGES]
 22 return ''

 It is also translating the given language name.  My suggestion is to
update the documentation saying that the developer should use ugettext_lazy
for the language name if they want it translated *or* at least use a
unicode string.

-- 
Henrique Romano

In the face of ambiguity, refuse the temptation to guess.
-- Tim Peters

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CA%2BEHudJ3%2B8tRpiEn_-%3DmnHOE1dOXXHVNzvhTEW4jOGQrmBxTqg%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Enforce the use of a unicode string in settings.LANGUAGES

[Henrique Romano]

On Wed, Jan 22, 2014 at 12:43 PM, Ramiro Morales  wrote:

> You aren't telling us all the story. there are many missing parts in your
> description of the issue you are finding so far so I don't think it's right
> to jump straight to the  "As per the documentation, it is not clear that
> you _must_ use a unicode string for the language name. ... What do you guys
> think about making it clear that the user should always use an unicode
> string for the LANGUAGES setting?" conclusion.
>
> Some questions:
>
> - What are you trying to do? Reducing the choics in the LANGUAGES setting?
> Do you intend to translate them from Portuguese to other laguages?
>

I'm using django-cms, I want to support only two languages in my system:
English and Portuguese.

- Why are you using gettext() at the module level? You should use
> ugettext_lazy() or ugettex_noop()
>

That was just an example, ugettext_* returns a functional.proxy, which
doesn't try to render the string when you print it on the screen, but if
you try to do it you will see the same result.

- Why did you show us a python interactive session instead of a Python
> soruce code file? How do you thing the interpreter can deduce the encoding
> of a bare string in that case?
>

I didn't care about the correct encoding, I used the interactive session
just to show that a _unicode_ string is necessary for ugettext.


> - Why in both cases are you using Django without are for setting up the
> settings infrastructure first?
>
> - Is the foofoo.py file are you (or rather you text editor) actually using
> the utf-8 encoding?
>
> Yes.

You link to the development documentation but the tests I performed below
> are against the latest 1.6.x stable branch code with Python 2.7.3 (another
> piece of information you don't give us):
>
> $ django-admin.py startproject lang_i18n
> $ cd lang_i18n/
> $ cat foofoo.py
> #  -*- coding: utf-8 -*-
>
> from django.utils.translation import ugettext, ugettext_lazy, ugettext_noop
>
> print(ugettext_lazy("Portugués"))
>
> $ file foofoo.py
> foofoo.py: Python script, UTF-8 Unicode text executable
>
> ramiro@mang:~/dtest/lang_i18n$ DJANGO_SETTINGS_MODULE=lang_i18n.settings
> python foofoo.py
> 
>
So the (lazy)  translation machinery is working.
>

As I wrote above, it "works" because it is returning a functional.proxy
object, if you try to print this object as string, you will see the same
error.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CA%2BEHud%2BDqchJmji7rzNeD3nebLeavArnWDhXNtXNA1KfYNkJBA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Enforce the use of a unicode string in settings.LANGUAGES

[Shai Berger]

On Wednesday 22 January 2014 18:29:18 Henrique Romano wrote:
> On Wed, Jan 22, 2014 at 2:16 PM, Shai Berger  wrote:
> > Wait -- so the real context (which, as Ramiro noted, you left out) is
> > 
> > # settings.py
> > 
> > LANGUAGES = (('pt_BR', _("Português")),)
> > 
> > Is it? Or is it
> > 
> > LANGUAGES = (('pt_BR', "Português"),)
> > 
> > If it is the former, then this is a generic issue of translatable strings
> > --
> > nothing to do with settings.LANGUAGES. It is usually assumed that, if you
> > are
> > making a string translatable, you write it in English -- then it's ASCII
> > and
> > all's well.
> > 
> > If it is the latter, please provide more details about the specific error
> > you
> > encountered (stack traces etc).
> 
> > In my test, LANGUAGES is defined as follows:
> LANGUAGES = [
> ('en-us', 'English'),
> ('pt-br', 'Português'),
> ]
> 
> As the documentation didn't make it clear that the language names should be
> unicode, I just used the translated name ("Português") in a plain string, I
> thought it would be OK since settings.py has the encoding declared at the
> top of the file.
> 
> As for the error, it happened inside django-cms.  There's a code that is
> called for building a list of the languages enabled and their names:
> 
> # cms/utils/i18n.py
>  19 def get_languages(site_id=None):
>  20 site_id = get_site(site_id)
>  21 result = get_cms_setting('LANGUAGES').get(site_id)
>  22 if not result:
>  23 result = []
>  24 defaults = get_cms_setting('LANGUAGES').get('default', {})
>  25 for code, name in settings.LANGUAGES:
>  26 lang = {'code': code, 'name': _(name)}
>  27 lang.update(defaults)
>  28 result.append(lang)
>  29 get_cms_setting('LANGUAGES')[site_id] = result
>  30 return result
> 
> In the code above it is trying to translate the language name and it is
> resulting in an error since the name isn't an unicode string.  I understand
> that this is happening in django-cms, but I still think it is relevant to
> document either that the name should be a unicode string or at least the
> english name of the language.

I don't think Django should take responsibility for a 3rd-party package which 
decides that some part of a setting should be translatable whether the user 
said so or not.

You might want to take this up with django-cms.

Shai.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/201401221838.24696.shai%40platonix.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Enforce the use of a unicode string in settings.LANGUAGES

[Henrique Romano]

On Wed, Jan 22, 2014 at 2:16 PM, Shai Berger  wrote:
>
> Wait -- so the real context (which, as Ramiro noted, you left out) is
>
> # settings.py
>
> LANGUAGES = (('pt_BR', _("Português")),)
>
> Is it? Or is it
>
> LANGUAGES = (('pt_BR', "Português"),)
>
> If it is the former, then this is a generic issue of translatable strings
> --
> nothing to do with settings.LANGUAGES. It is usually assumed that, if you
> are
> making a string translatable, you write it in English -- then it's ASCII
> and
> all's well.
>
> If it is the latter, please provide more details about the specific error
> you
> encountered (stack traces etc).
>
> In my test, LANGUAGES is defined as follows:

LANGUAGES = [
('en-us', 'English'),
('pt-br', 'Português'),
]

As the documentation didn't make it clear that the language names should be
unicode, I just used the translated name ("Português") in a plain string, I
thought it would be OK since settings.py has the encoding declared at the
top of the file.

As for the error, it happened inside django-cms.  There's a code that is
called for building a list of the languages enabled and their names:

# cms/utils/i18n.py
 19 def get_languages(site_id=None):
 20 site_id = get_site(site_id)
 21 result = get_cms_setting('LANGUAGES').get(site_id)
 22 if not result:
 23 result = []
 24 defaults = get_cms_setting('LANGUAGES').get('default', {})
 25 for code, name in settings.LANGUAGES:
 26 lang = {'code': code, 'name': _(name)}
 27 lang.update(defaults)
 28 result.append(lang)
 29 get_cms_setting('LANGUAGES')[site_id] = result
 30 return result

In the code above it is trying to translate the language name and it is
resulting in an error since the name isn't an unicode string.  I understand
that this is happening in django-cms, but I still think it is relevant to
document either that the name should be a unicode string or at least the
english name of the language.

-- 
Henrique Romano

In the face of ambiguity, refuse the temptation to guess.
-- Tim Peters

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CA%2BEHudJe91RoJoGBV14u49zW_h70v6drf336giyWA-Zh%3DGMHsQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Enforce the use of a unicode string in settings.LANGUAGES

[Shai Berger]

On Wednesday 22 January 2014 18:01:14 Henrique Romano wrote:
> On Wed, Jan 22, 2014 at 12:39 PM, Shai Berger  wrote:
> > This has nothing to do with the LANGUAGES setting, or the string being a
> > language name. it just so happens that ugettext tries to return unicode,
> > and
> > so for an untranslated string s it returns unicode(s). You can get the
> > same error by writing
> > 
> > unicode("Português")
> 
> Correct.
> 
> > You should make sure that every string you pass to unicode(), directly or
> > indirectly, is either a unicode object or an ASCII-only string (except in
> > cases where you also pass the encoding); but that is general Python, not
> > Django-specific.
> 
> The point is that I didn't know I had to pass an unicode string, I didn't
> even know about the internals (of django-cms or django, I don't remember)
> where the language name is being translated and therefore an unicode string
> was necessary.  That's what I'm suggesting here, to make it explicit that
> the language name string must be a unicode string.

Wait -- so the real context (which, as Ramiro noted, you left out) is

# settings.py

LANGUAGES = (('pt_BR', _("Português")),)

Is it? Or is it

LANGUAGES = (('pt_BR', "Português"),)

If it is the former, then this is a generic issue of translatable strings -- 
nothing to do with settings.LANGUAGES. It is usually assumed that, if you are 
making a string translatable, you write it in English -- then it's ASCII and 
all's well.

If it is the latter, please provide more details about the specific error you 
encountered (stack traces etc).

Shai.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/201401221816.59089.shai%40platonix.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Enforce the use of a unicode string in settings.LANGUAGES

[Henrique Romano]

On Wed, Jan 22, 2014 at 12:39 PM, Shai Berger  wrote:

>
> This has nothing to do with the LANGUAGES setting, or the string being a
> language name. it just so happens that ugettext tries to return unicode,
> and
> so for an untranslated string s it returns unicode(s). You can get the same
> error by writing
>
> unicode("Português")
>

Correct.


> You should make sure that every string you pass to unicode(), directly or
> indirectly, is either a unicode object or an ASCII-only string (except in
> cases where you also pass the encoding); but that is general Python, not
> Django-specific.
>

The point is that I didn't know I had to pass an unicode string, I didn't
even know about the internals (of django-cms or django, I don't remember)
where the language name is being translated and therefore an unicode string
was necessary.  That's what I'm suggesting here, to make it explicit that
the language name string must be a unicode string.

-- 
Henrique Romano

In the face of ambiguity, refuse the temptation to guess.
-- Tim Peters

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CA%2BEHud%2BVhKELjNt-TL1zL5smiszXTMe5LGQ2TFODJWC9Efe%3Dpw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Enforce the use of a unicode string in settings.LANGUAGES

[gilberto dos santos alves]

yes i will try with django 1.6.


2014/1/22 Henrique Romano 

> On Wed, Jan 22, 2014 at 11:38 AM, gilberto dos santos alves <
> gsa...@gmail.com> wrote:
>
>> please look details about on [1]. if you put
>>
>> # -*- coding: utf-8 -*-  on sources and config files for django your string 
>> "português" will be automatically handled.
>>
>>
> Can you just try what I reported?  For example:
>
> $ cat ~/foofoo.py
> # -*- coding: utf-8 -*-
> from django.utils.translation import ugettext
>
> print ugettext("Português")
> $ python ~/foofoo.py
> Traceback (most recent call last):
> ...
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 7:
> ordinal not in range(128)
>
> So, I'm not sure what you are talking about.
> --
> Henrique Romano
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CA%2BEHudKiJcgLQg-K2pwQrBFFcCTr%2BpV_t9VqmkSo%3D6n-6Kjqaw%40mail.gmail.com
> .
>
> For more options, visit https://groups.google.com/groups/opt_out.
>



-- 
gilberto dos santos alves
+55.11.98646-5049
sao paulo - sp - brasil

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAP9G-NLgyqk%2ByUtSaZ6ueTbLiFfb2xLVCrT8n2Si7%3DgxKE626A%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Enforce the use of a unicode string in settings.LANGUAGES

[Ramiro Morales]

On Wed, Jan 22, 2014 at 10:59 AM, Henrique Romano wrote:

> On Wed, Jan 22, 2014 at 11:38 AM, gilberto dos santos alves <
> gsa...@gmail.com> wrote:
>
>> please look details about on [1]. if you put
>>
>> # -*- coding: utf-8 -*-  on sources and config files for django your string 
>> "português" will be automatically handled.
>>
>>
> Can you just try what I reported?  For example:
>
> $ cat ~/foofoo.py
> # -*- coding: utf-8 -*-
> from django.utils.translation import ugettext
>
> print ugettext("Português")
> $ python ~/foofoo.py
> Traceback (most recent call last):
> ...
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 7:
> ordinal not in range(128)
>
> So, I'm not sure what you are talking about.
>

You aren't telling us all the story. there are many missing parts in your
description of the issue you are finding so far so I don't think it's right
to jump straight to the  "As per the documentation, it is not clear that
you _must_ use a unicode string for the language name. ... What do you guys
think about making it clear that the user should always use an unicode
string for the LANGUAGES setting?" conclusion.

Some questions:

- What are you trying to do? Reducing the choics in the LANGUAGES setting?
Do you intend to translate them from Portuguese to other laguages?

- Why are you using gettext() at the module level? You should use
ugettext_lazy() or ugettex_noop()

- Why did you show us a python interactive session instead of a Python
soruce code file? How do you thing the interpreter can deduce the encoding
of a bare string in that case?

- Why in both cases are you using Django without are for setting up the
settings infrastructure first?

- Is the foofoo.py file are you (or rather you text editor) actually using
the utf-8 encoding?

You link to the development documentation but the tests I performed below
are against the latest 1.6.x stable branch code with Python 2.7.3 (another
piece of information you don't give us):

$ django-admin.py startproject lang_i18n
$ cd lang_i18n/
$ cat foofoo.py
#  -*- coding: utf-8 -*-

from django.utils.translation import ugettext, ugettext_lazy, ugettext_noop

print(ugettext_lazy("Portugués"))

$ file foofoo.py
foofoo.py: Python script, UTF-8 Unicode text executable

ramiro@mang:~/dtest/lang_i18n$ DJANGO_SETTINGS_MODULE=lang_i18n.settings
python foofoo.py


So the (lazy)  translation machinery is working.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAO7PdF9NgfG35MvJ5X24M7cSb%3DuOmZLkgkR9250DuT7ESCZO%3Dw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Enforce the use of a unicode string in settings.LANGUAGES

[Shai Berger]

On Wednesday 22 January 2014 15:59:23 Henrique Romano wrote:
> On Wed, Jan 22, 2014 at 11:38 AM, gilberto dos santos alves <
> 
> gsa...@gmail.com> wrote:
> > please look details about on [1]. if you put
> > 
> > # -*- coding: utf-8 -*-  on sources and config files for django your
> > string "português" will be automatically handled.
> 
> Can you just try what I reported?  For example:
> 
> $ cat ~/foofoo.py
> # -*- coding: utf-8 -*-
> from django.utils.translation import ugettext
> 
> print ugettext("Português")
> $ python ~/foofoo.py
> Traceback (most recent call last):
> ...
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 7:
> ordinal not in range(128)
> 
> So, I'm not sure what you are talking about.

This has nothing to do with the LANGUAGES setting, or the string being a 
language name. it just so happens that ugettext tries to return unicode, and 
so for an untranslated string s it returns unicode(s). You can get the same 
error by writing

unicode("Português")

You should make sure that every string you pass to unicode(), directly or 
indirectly, is either a unicode object or an ASCII-only string (except in 
cases where you also pass the encoding); but that is general Python, not 
Django-specific.

As far as file headers are concerned, you may want to use

from __future__ import unicode_literals

which really does make all your strings unicode.

Please take further discussion of this to django-users or other forums.

HTH,
Shai.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/201401221639.43305.shai%40platonix.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Enforce the use of a unicode string in settings.LANGUAGES

[Henrique Romano]

On Wed, Jan 22, 2014 at 11:38 AM, gilberto dos santos alves <
gsa...@gmail.com> wrote:

> please look details about on [1]. if you put
>
> # -*- coding: utf-8 -*-  on sources and config files for django your string 
> "português" will be automatically handled.
>
>
Can you just try what I reported?  For example:

$ cat ~/foofoo.py
# -*- coding: utf-8 -*-
from django.utils.translation import ugettext

print ugettext("Português")
$ python ~/foofoo.py
Traceback (most recent call last):
...
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 7:
ordinal not in range(128)

So, I'm not sure what you are talking about.
-- 
Henrique Romano

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CA%2BEHudKiJcgLQg-K2pwQrBFFcCTr%2BpV_t9VqmkSo%3D6n-6Kjqaw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Enforce the use of a unicode string in settings.LANGUAGES

[gilberto dos santos alves]

please look details about on [1]. if you put

# -*- coding: utf-8 -*-  on sources and config files for django your
string "português" will be automatically handled.

regards.


[1] http://docs.python.org/2/howto/unicode.html


2014/1/22 Henrique Romano 

> Hi,
>
> On Wed, Jan 22, 2014 at 12:37 AM, gilberto dos santos alves <
> gsa...@gmail.com> wrote:
>
>> please see that it is python directive not django. for all sources it is
>> a good practive for all we that use pt-br utf-8 explicit this on second
>> line of file python code
>>
>> # -*- coding: utf-8 -*-
>>
>>
> Specyfing the encoding of the file don't make strings unicode
> automatically, therefore doing it won't solve the problem.  I think the
> problem is still relevant.
>
> Thanks
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CA%2BEHudJHqD4w%2B3ByRq8v4Mu6wbQ2DR0LqvNOJUyLfU_V8fZkGA%40mail.gmail.com
> .
>
> For more options, visit https://groups.google.com/groups/opt_out.
>



-- 
gilberto dos santos alves
+55.11.98646-5049
sao paulo - sp - brasil

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAP9G-NLpjBRfs83NvQ3gCYtD2NuR6Ye8DgHbKd%2B0OynGM3RXAw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Enforce the use of a unicode string in settings.LANGUAGES

[Henrique Romano]

Hi,

On Wed, Jan 22, 2014 at 12:37 AM, gilberto dos santos alves <
gsa...@gmail.com> wrote:

> please see that it is python directive not django. for all sources it is a
> good practive for all we that use pt-br utf-8 explicit this on second line
> of file python code
>
> # -*- coding: utf-8 -*-
>
>
Specyfing the encoding of the file don't make strings unicode
automatically, therefore doing it won't solve the problem.  I think the
problem is still relevant.

Thanks

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CA%2BEHudJHqD4w%2B3ByRq8v4Mu6wbQ2DR0LqvNOJUyLfU_V8fZkGA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Enforce the use of a unicode string in settings.LANGUAGES

[gilberto dos santos alves]

please see that it is python directive not django. for all sources it is a
good practive for all we that use pt-br utf-8 explicit this on second line
of file python code

# -*- coding: utf-8 -*-

regards!




2014/1/21 Henrique Romano 

> Hi,
>
> As per the documentation[1], it is not clear that you _must_ use a unicode
> string for the language name.  If you don't use an unicode string, the
> following can happen:
>
> >>> from django.utils.translation import ugettext
> >>> ugettext("Português")
> Traceback (most recent call last):
> ...
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 7:
> ordinal not in range(128)
>
> As opposed to:
>
> >>> ugettext(u"Português")
> u'Portugu\xeas'
>
> I was just having an issue where the languages available was being
> rendered in the template, but since TEMPLATE_DEBUG was enabled, no errors
> was generated in the development environment.  Switching TEMPLATE_DEBUG off
> in the production resulted in an exception with almost no clue on what
> happened.
>
> What do you guys think about making it clear that the user should always
> use an unicode string for the LANGUAGES setting?
>
> [1] https://docs.djangoproject.com/en/dev/ref/settings/#languages
> --
> Henrique Romano
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CA%2BEHudKfPXxry6T2tW_6ZNFzJgAJUHs3nO_mLkdp3Xk2wS09xg%40mail.gmail.com
> .
> For more options, visit https://groups.google.com/groups/opt_out.
>



-- 
gilberto dos santos alves
+55.11.98646-5049
sao paulo - sp - brasil

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAP9G-NLKRTiyM_P-1aMR2dKZh%3DgNU_0s6GYGoXLO%2BjmzZYnsCw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Enforce the use of a unicode string in settings.LANGUAGES

[Henrique Romano]

Hi,

As per the documentation[1], it is not clear that you _must_ use a unicode
string for the language name.  If you don't use an unicode string, the
following can happen:

>>> from django.utils.translation import ugettext
>>> ugettext("Português")
Traceback (most recent call last):
...
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 7:
ordinal not in range(128)

As opposed to:

>>> ugettext(u"Português")
u'Portugu\xeas'

I was just having an issue where the languages available was being rendered
in the template, but since TEMPLATE_DEBUG was enabled, no errors was
generated in the development environment.  Switching TEMPLATE_DEBUG off in
the production resulted in an exception with almost no clue on what
happened.

What do you guys think about making it clear that the user should always
use an unicode string for the LANGUAGES setting?

[1] https://docs.djangoproject.com/en/dev/ref/settings/#languages
-- 
Henrique Romano

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CA%2BEHudKfPXxry6T2tW_6ZNFzJgAJUHs3nO_mLkdp3Xk2wS09xg%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.