Re: [Toolserver-l] Encoding issue using SGE

[Patricia Pintilie]

Your welcome if I had any part in helping you out.
On May 23, 2013 2:31 PM, "Hercule Hercule" 
wrote:

> The addition of console_encoding = 'utf-8' in my user-config.py solved
> the issue. Thanks everybody for your help
>
>
> 2013/5/23 Hercule Hercule 
>
>> I added
>> export LANG
>> LANG=fr_FR.UTF-8
>> in the script, but it still make incorrect comments.
>>
>> I add enclosed the sh file, if it can help
>>
>>
>> 2013/5/23 Lars Aronsson 
>>
>>> On 05/23/2013 02:40 PM, Hercule Hercule wrote:
>>>
 I already tried to add

 export LANG=fr_FR.UTF-8 but I then have an alert : LANG=fr_FR.UTF-8 is
 not an identifier
 LANG=fr_FR.UTF-8 at the top of the shell don't change anything.

>>>
>>> The syntax "export A=B" is from bash, a common login shell.
>>> However, most scripts are traditionally written in #!/bin/sh
>>> where the syntax requires "export A" and "A=B" to be written
>>> as two separate statements.
>>>
>>> (Knowledge from the 1980s.)
>>>
>>>
>>> --
>>>   Lars Aronsson (l...@aronsson.se)
>>>   Aronsson Datateknik - http://aronsson.se
>>>
>>>
>>>
>>>
>>> __**_
>>> Toolserver-l mailing list 
>>> (Toolserver-l@lists.wikimedia.**org
>>> )
>>> https://lists.wikimedia.org/**mailman/listinfo/toolserver-l
>>> Posting guidelines for this list: https://wiki.toolserver.org/**
>>> view/Mailing_list_etiquette
>>>
>>
>>
>
> ___
> Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/toolserver-l
> Posting guidelines for this list:
> https://wiki.toolserver.org/view/Mailing_list_etiquette
>
___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Encoding issue using SGE

[Hercule Hercule]

The addition of console_encoding = 'utf-8' in my user-config.py solved the
issue. Thanks everybody for your help


2013/5/23 Hercule Hercule 

> I added
> export LANG
> LANG=fr_FR.UTF-8
> in the script, but it still make incorrect comments.
>
> I add enclosed the sh file, if it can help
>
>
> 2013/5/23 Lars Aronsson 
>
>> On 05/23/2013 02:40 PM, Hercule Hercule wrote:
>>
>>> I already tried to add
>>>
>>> export LANG=fr_FR.UTF-8 but I then have an alert : LANG=fr_FR.UTF-8 is
>>> not an identifier
>>> LANG=fr_FR.UTF-8 at the top of the shell don't change anything.
>>>
>>
>> The syntax "export A=B" is from bash, a common login shell.
>> However, most scripts are traditionally written in #!/bin/sh
>> where the syntax requires "export A" and "A=B" to be written
>> as two separate statements.
>>
>> (Knowledge from the 1980s.)
>>
>>
>> --
>>   Lars Aronsson (l...@aronsson.se)
>>   Aronsson Datateknik - http://aronsson.se
>>
>>
>>
>>
>> __**_
>> Toolserver-l mailing list 
>> (Toolserver-l@lists.wikimedia.**org
>> )
>> https://lists.wikimedia.org/**mailman/listinfo/toolserver-l
>> Posting guidelines for this list: https://wiki.toolserver.org/**
>> view/Mailing_list_etiquette
>>
>
>
___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Encoding issue using SGE

[Hercule Hercule]

I added
export LANG
LANG=fr_FR.UTF-8
in the script, but it still make incorrect comments.

I add enclosed the sh file, if it can help

2013/5/23 Lars Aronsson 

> On 05/23/2013 02:40 PM, Hercule Hercule wrote:
>
>> I already tried to add
>>
>> export LANG=fr_FR.UTF-8 but I then have an alert : LANG=fr_FR.UTF-8 is
>> not an identifier
>> LANG=fr_FR.UTF-8 at the top of the shell don't change anything.
>>
>
> The syntax "export A=B" is from bash, a common login shell.
> However, most scripts are traditionally written in #!/bin/sh
> where the syntax requires "export A" and "A=B" to be written
> as two separate statements.
>
> (Knowledge from the 1980s.)
>
>
> --
>   Lars Aronsson (l...@aronsson.se)
>   Aronsson Datateknik - http://aronsson.se
>
>
>
>
> __**_
> Toolserver-l mailing list 
> (Toolserver-l@lists.wikimedia.**org
> )
> https://lists.wikimedia.org/**mailman/listinfo/toolserver-l
> Posting guidelines for this list: https://wiki.toolserver.org/**
> view/Mailing_list_etiquette
>


LienFr.sh
Description: Bourne shell script
___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Encoding issue using SGE

[Lars Aronsson]

On 05/23/2013 02:40 PM, Hercule Hercule wrote:

I already tried to add

export LANG=fr_FR.UTF-8 but I then have an alert : LANG=fr_FR.UTF-8 is 
not an identifier

LANG=fr_FR.UTF-8 at the top of the shell don't change anything.


The syntax "export A=B" is from bash, a common login shell.
However, most scripts are traditionally written in #!/bin/sh
where the syntax requires "export A" and "A=B" to be written
as two separate statements.

(Knowledge from the 1980s.)


--
  Lars Aronsson (l...@aronsson.se)
  Aronsson Datateknik - http://aronsson.se



___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Encoding issue using SGE

[Merlijn van Deen]

I just checked what locale returns under SGE, and this is en_US.UTF-8
- so setting locale should not change - so it's not really clear to me
why this error would occur.

In any case, you can try setting

console_encoding = 'utf-8'

in your user-config.py. That should take care of the problem,
irrespective of locale setting.

Merlijn

On 23 May 2013 14:40, Hercule Hercule  wrote:
> I already tried to add
>
> export LANG=fr_FR.UTF-8 but I then have an alert : LANG=fr_FR.UTF-8 is not
> an identifier
> LANG=fr_FR.UTF-8 at the top of the shell don't change anything.
>
> What I don't understand is that the problem only comes with SGE submit. And
> the command 'locale' in the shell shows en_US.UTF-8 for every property
>
> I will try LANG=fr_FR.UTF-8 python /home/hercule/template.py this evening
>
>
> Le 23 mai 2013 09:16, Merlijn van Deen  a écrit :
>>
>> Hi Hercule,
>>
>> On 23 May 2013 00:38, Hercule Hercule  wrote:
>> > python /home/hercule/template.py "Lien" "Lien/Conversion automatique"
>> > -cat:"Page utilisant Lien pour un article existant" -summary:"[Bot] :
>> > transformation de liens avec le modèle {{Lien}} en lien interne, suite à
>> > la
>> > création de l'article correspondant" -assubst -pt:0 -always
>> > mutt -s "Fin de LienFr" hercule < /home/hercule/logs/LienFr.log
>>
>>
>> If you have saved the file in the utf-8 encoding (which the edit
>> summary suggests), try using (on one line)
>>
>> LANG=en_GB.UTF-8 python /home/hercule/template.py ...
>>
>> or (on two lines)
>> export LANG=en_GB.UTF-8
>> python /home/hercule/template.py ...
>>
>> Basically, you have to tell pywikipedia which encoding you have used
>> for the shell arguments - in your case UTF-8. Otherwise it will use
>> latin-1 as default, which results in the mojibake you saw.
>>
>> Merlijn
>>
>> ___
>> Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
>> https://lists.wikimedia.org/mailman/listinfo/toolserver-l
>> Posting guidelines for this list:
>> https://wiki.toolserver.org/view/Mailing_list_etiquette
>
>
>
> ___
> Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/toolserver-l
> Posting guidelines for this list:
> https://wiki.toolserver.org/view/Mailing_list_etiquette

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Encoding issue using SGE

[Hercule Hercule]

I already tried to add

export LANG=fr_FR.UTF-8 but I then have an alert : LANG=fr_FR.UTF-8 is not
an identifier
LANG=fr_FR.UTF-8 at the top of the shell don't change anything.

What I don't understand is that the problem only comes with SGE submit. And
the command 'locale' in the shell shows en_US.UTF-8 for every property

I will try LANG=fr_FR.UTF-8 python /home/hercule/template.py this evening


Le 23 mai 2013 09:16, Merlijn van Deen  a écrit :

> Hi Hercule,
>
> On 23 May 2013 00:38, Hercule Hercule  wrote:
> > python /home/hercule/template.py "Lien" "Lien/Conversion automatique"
> > -cat:"Page utilisant Lien pour un article existant" -summary:"[Bot] :
> > transformation de liens avec le modèle {{Lien}} en lien interne, suite à
> la
> > création de l'article correspondant" -assubst -pt:0 -always
> > mutt -s "Fin de LienFr" hercule < /home/hercule/logs/LienFr.log
>
>
> If you have saved the file in the utf-8 encoding (which the edit
> summary suggests), try using (on one line)
>
> LANG=en_GB.UTF-8 python /home/hercule/template.py ...
>
> or (on two lines)
> export LANG=en_GB.UTF-8
> python /home/hercule/template.py ...
>
> Basically, you have to tell pywikipedia which encoding you have used
> for the shell arguments - in your case UTF-8. Otherwise it will use
> latin-1 as default, which results in the mojibake you saw.
>
> Merlijn
>
> ___
> Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/toolserver-l
> Posting guidelines for this list:
> https://wiki.toolserver.org/view/Mailing_list_etiquette
>
___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Encoding issue using SGE

[Patricia Pintilie]

General chat • Re: Navigating Multiboot GRUB2 menu entries successfully.
http://forum.porteus.org/viewtopic.php?t=2195&p=15042#p15042
On May 23, 2013 3:12 AM, "Merlijn van Deen"  wrote:

> Hi Hercule,
>
> On 23 May 2013 00:38, Hercule Hercule  wrote:
> > python /home/hercule/template.py "Lien" "Lien/Conversion automatique"
> > -cat:"Page utilisant Lien pour un article existant" -summary:"[Bot] :
> > transformation de liens avec le modèle {{Lien}} en lien interne, suite à
> la
> > création de l'article correspondant" -assubst -pt:0 -always
> > mutt -s "Fin de LienFr" hercule < /home/hercule/logs/LienFr.log
>
>
> If you have saved the file in the utf-8 encoding (which the edit
> summary suggests), try using (on one line)
>
> LANG=en_GB.UTF-8 python /home/hercule/template.py ...
>
> or (on two lines)
> export LANG=en_GB.UTF-8
> python /home/hercule/template.py ...
>
> Basically, you have to tell pywikipedia which encoding you have used
> for the shell arguments - in your case UTF-8. Otherwise it will use
> latin-1 as default, which results in the mojibake you saw.
>
> Merlijn
>
> ___
> Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/toolserver-l
> Posting guidelines for this list:
> https://wiki.toolserver.org/view/Mailing_list_etiquette
___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Encoding issue using SGE

[Merlijn van Deen]

Hi Hercule,

On 23 May 2013 00:38, Hercule Hercule  wrote:
> python /home/hercule/template.py "Lien" "Lien/Conversion automatique"
> -cat:"Page utilisant Lien pour un article existant" -summary:"[Bot] :
> transformation de liens avec le modèle {{Lien}} en lien interne, suite à la
> création de l'article correspondant" -assubst -pt:0 -always
> mutt -s "Fin de LienFr" hercule < /home/hercule/logs/LienFr.log


If you have saved the file in the utf-8 encoding (which the edit
summary suggests), try using (on one line)

LANG=en_GB.UTF-8 python /home/hercule/template.py ...

or (on two lines)
export LANG=en_GB.UTF-8
python /home/hercule/template.py ...

Basically, you have to tell pywikipedia which encoding you have used
for the shell arguments - in your case UTF-8. Otherwise it will use
latin-1 as default, which results in the mojibake you saw.

Merlijn

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Encoding issue using SGE

[Hercule Hercule]

Hello,

If I don't have the three first lines, the log give

newtask: exec of /var/sge/spool/execd/wolfsbane/job_scripts/2016385 failed:
No such file or directory

When making the iconv I have no difference at all. I tried using LienFr and
LienFr2 with the same result.

Regards

Hercule


2013/5/23 Platonides 

> On 23/05/13 00:38, Hercule Hercule wrote:
> > Hello,
> >
> > I deleted one of my script, and try to recreate it. But I now have an
> > encoding issue when I run it with SGE. I searched a solution during all
> > this evening and can't find any idea to solve it. I hope one of you will
> > be able to help me.
> >
> > I created the file /home/hercule/batch/LienFr.sh with this content :
> >
> > #-*- coding: utf-8 -*-
> > #!bin/bash
> > #! /usr/bin/python
>
> These shebangs are crazy. You would want just "#!/bin/bash" as the first
> line.
>
>
> > #$ -l h_rt=1:30:00
> > #$ -l virtual_free=500M
> > #$ -l arch='*'
> > #$ -m as
> > #$ -o /home/hercule/logs/LienFr.log
> > #$ -j y
> > #$ -wd /home/hercule/
> > python /home/hercule/template.py "Lien" "Lien/Conversion automatique"
> > -cat:"Page utilisant Lien pour un article existant" -summary:"[Bot] :
> > transformation de liens avec le modèle {{Lien}} en lien interne, suite à
> > la création de l'article correspondant" -assubst -pt:0 -always
> > mutt -s "Fin de LienFr" hercule < /home/hercule/logs/LienFr.log
> >
> >
> > When I simply run /home/hercule/batch/LienFr.sh in the terminal the
> > result is correct :
> >
> http://fr.wikipedia.org/w/index.php?title=Uttar_Pradesh&diff=93370887&oldid=93370870
> >
> > But when I use the command line
> > qcronsub -N LienFr /home/hercule/batch/LienFr.sh
> >
> > I have a problem with the summary :
> >
> >
> http://fr.wikipedia.org/w/index.php?title=Uttar_Pradesh&diff=93370948&oldid=93370932
> >
> > Do you know how to solve this ?
> >
> > Regards
> >
> > Hercule
>
> Seems the string was converted to utf-8 twice. What did you use to edit
> the file?
>
> Try converting it with iconv:
> iconv -f utf-8 -t iso-8859-1 < LienFr.sh > LienFr2.sh
>
>
>
> ___
> Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/toolserver-l
> Posting guidelines for this list:
> https://wiki.toolserver.org/view/Mailing_list_etiquette
___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Encoding issue using SGE

[Platonides]

On 23/05/13 00:38, Hercule Hercule wrote:
> Hello,
> 
> I deleted one of my script, and try to recreate it. But I now have an
> encoding issue when I run it with SGE. I searched a solution during all
> this evening and can't find any idea to solve it. I hope one of you will
> be able to help me.
> 
> I created the file /home/hercule/batch/LienFr.sh with this content :
> 
> #-*- coding: utf-8 -*-
> #!bin/bash
> #! /usr/bin/python

These shebangs are crazy. You would want just "#!/bin/bash" as the first
line.


> #$ -l h_rt=1:30:00
> #$ -l virtual_free=500M
> #$ -l arch='*'
> #$ -m as
> #$ -o /home/hercule/logs/LienFr.log
> #$ -j y
> #$ -wd /home/hercule/
> python /home/hercule/template.py "Lien" "Lien/Conversion automatique"
> -cat:"Page utilisant Lien pour un article existant" -summary:"[Bot] :
> transformation de liens avec le modèle {{Lien}} en lien interne, suite à
> la création de l'article correspondant" -assubst -pt:0 -always
> mutt -s "Fin de LienFr" hercule < /home/hercule/logs/LienFr.log
> 
> 
> When I simply run /home/hercule/batch/LienFr.sh in the terminal the
> result is correct :
> http://fr.wikipedia.org/w/index.php?title=Uttar_Pradesh&diff=93370887&oldid=93370870
> 
> But when I use the command line 
> qcronsub -N LienFr /home/hercule/batch/LienFr.sh
> 
> I have a problem with the summary :
> 
> http://fr.wikipedia.org/w/index.php?title=Uttar_Pradesh&diff=93370948&oldid=93370932
> 
> Do you know how to solve this ?
> 
> Regards
> 
> Hercule

Seems the string was converted to utf-8 twice. What did you use to edit
the file?

Try converting it with iconv:
iconv -f utf-8 -t iso-8859-1 < LienFr.sh > LienFr2.sh



___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

[Toolserver-l] Encoding issue using SGE

[Hercule Hercule]

Hello,

I deleted one of my script, and try to recreate it. But I now have an
encoding issue when I run it with SGE. I searched a solution during all
this evening and can't find any idea to solve it. I hope one of you will be
able to help me.

I created the file /home/hercule/batch/LienFr.sh with this content :

#-*- coding: utf-8 -*-
#!bin/bash
#! /usr/bin/python
#$ -l h_rt=1:30:00
#$ -l virtual_free=500M
#$ -l arch='*'
#$ -m as
#$ -o /home/hercule/logs/LienFr.log
#$ -j y
#$ -wd /home/hercule/
python /home/hercule/template.py "Lien" "Lien/Conversion automatique"
-cat:"Page utilisant Lien pour un article existant" -summary:"[Bot] :
transformation de liens avec le modèle {{Lien}} en lien interne, suite à la
création de l'article correspondant" -assubst -pt:0 -always
mutt -s "Fin de LienFr" hercule < /home/hercule/logs/LienFr.log


When I simply run /home/hercule/batch/LienFr.sh in the terminal the result
is correct :
http://fr.wikipedia.org/w/index.php?title=Uttar_Pradesh&diff=93370887&oldid=93370870

But when I use the command line
qcronsub -N LienFr /home/hercule/batch/LienFr.sh

I have a problem with the summary :

http://fr.wikipedia.org/w/index.php?title=Uttar_Pradesh&diff=93370948&oldid=93370932


Do you know how to solve this ?

Regards

Hercule
___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette