file.write() of non-ASCII characters differs in Interpreted Python than in script run
Dear All,
I experienced an incomprehensible behavior (I've spent already many hours on
this subject): the `file.write('string')` provides an error in run mode and not
when interpreted at the console. The string must contain non-ASCII characters.
If all ASCII, there is no error.
The following example shows what I can see. I must overlook something because I
cannot think Python makes a difference between interpreted and run modes and
yet ... Can someone please check that subject.
Thank you in advance.
René
Code extract from WSGI application (reply.py)
=
request_body = environ['wsgi.input'].read(request_body_size)# bytes
rb = request_body.decode() # string
d = parse_qs(rb)# dict
f = open('logbytes', 'ab')
g = open('logstr', 'a')
h = open('logdict', 'a')
f.write(request_body)
g.write(str(type(request_body)) + '\t' + str(type(rb)) + '\t' +
str(type(d)) + '\n')
h.write(str(d) + '\n') <--- line 28 of the application
h.close()
g.close()
f.close()
Tail of Apache2 error.log
=
[Tue Aug 25 20:24:04.657933 2015] [wsgi:error] [pid 3677:tid 3029764928]
[remote 192.168.1.5:27575] File "reply.py", line 28, in application
[Tue Aug 25 20:24:04.658001 2015] [wsgi:error] [pid 3677:tid 3029764928]
[remote 192.168.1.5:27575] h.write(str(d) + '\\n')
[Tue Aug 25 20:24:04.658201 2015] [wsgi:error] [pid 3677:tid 3029764928]
[remote 192.168.1.5:27575] UnicodeEncodeError: 'ascii' codec can't encode
character '\\xc7' in position 15: ordinal not in range(128)
Checking what has been logged
=
rse@Alibaba:~/test$ cat logbytes
userName=Ça va ! <--- this was indeed the input (notice the
french C + cedilla)
Unicode U+00C7ALT-0199UTF-8 C387
Reading the logbytes file one can verify
that Ç is indeed represented by the 2 bytes
\xC3 and \x87
rse@Alibaba:~/test$ cat logstr
rse@Alibaba:~/test$ cat logdict
rse@Alibaba:~/test$ <--- Obviously empty because of error
Trying similar code within the Python interpreter
=
rse@Alibaba:~/test$ python
Python 3.4.0 (default, Jun 19 2015, 14:18:46)
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> di = {'userName': ['Ça va !']}<--- A dictionary
>>> str(di)
"{'userName': ['Ça va !']}" <--- and its string representation
>>> type(str(di))
<--- Is a string indeed
>>> fi = open('essai', 'a')
>>> fi.write(str(di) + '\n')
26 <--- It works well
>>> fi.close()
>>>
Checking what has been written
==
rse@Alibaba:~/test$ cat essai
{'userName': ['Ça va !']} <--- The result is correct
rse@Alibaba:~/test$
No error if all ASCII
=
If the input is `userName=Rene` for instance then there is no error and the
`logdict' does indeed then contain the text of the dictionary
`{'userName': ['Rene']}`
--
https://mail.python.org/mailman/listinfo/python-list
Re: file.write() of non-ASCII characters differs in Interpreted Python than in script run
Dear Chris, I can confirm it is Python 3. Here is the line from the Apache2 log: [Wed Aug 26 10:28:01.508194 2015] [mpm_worker:notice] [pid 1120:tid 3074398848] AH00292: Apache/2.4.7 (Ubuntu) OpenSSL/1.0.1f mod_wsgi/4.4.13 Python/3.4.0 configured -- resuming normal operations As a matter of fact, I connect to the same machine that runs Apache2/mod_wsgi/Python via PuTTY and when I open the Python interpreter it responds: > Python 3.4.0 (default, Jun 19 2015, 14:18:46) > [GCC 4.8.2] on linux > Type "help", "copyright", "credits" or "license" for more information. Hence exactly the same Python 3.4.0 version. By the way my whole installation is defaulted to UTF-8: HTML: Javascript:
Re: file.write() of non-ASCII characters differs in Interpreted Python than in script run
Dear Chris, Thank you. I got the answer (at least a partial one) that I will share in a while. I will first respond to the other posts I received to thank each and everyone. Please stay tuned. René -- https://mail.python.org/mailman/listinfo/python-list
Re: file.write() of non-ASCII characters differs in Interpreted Python than in script run
Thank you Pete. Indeed it has to do with choice of encoding. I'll be back in a short while with more details. Cheers, René -- https://mail.python.org/mailman/listinfo/python-list
Re: file.write() of non-ASCII characters differs in Interpreted Python than in script run
Dear Dieter, Indeed there is a difference. I will share my discoveries in a while after I respond to each one. Be in touch soon. Thanks. René -- https://mail.python.org/mailman/listinfo/python-list
Re: file.write() of non-ASCII characters differs in Interpreted Python than in script run
Thank you Chris. I'll share my findings in a moment. Please bear with me a bit more time. Cheers, René -- https://mail.python.org/mailman/listinfo/python-list
Re: file.write() of non-ASCII characters differs in Interpreted Python than in script run
Dear All, First, thanks to each and everyone. There is indeed a solution by I haven't yet found the root of the problem (I'll come back to that at the end of my post). 1) After many trials and errors, I found that the problem was with the write() function in `h.write(str(d) + '\n')` and not with the argument itself which is a perfect string. 2) Reading the documentation it refers to the open() function and its preferred encoding. 3) I checked with the interpreter and got: rse@Alibaba:~/test$ python Python 3.4.0 (default, Jun 19 2015, 14:18:46) [GCC 4.8.2] on linux Type "help", "copyright", "credits" or "license" for more information. >>> from locale import * >>> getpreferredencoding() 'UTF-8' >>> 4) I knew everything was set up with UTF-8 (refer my first answer to Chris K.) so I couldn't believe it ! Another dead end ? 5) I had to make sure it was the same within the application, so I added a couple of statements to get and record the preferred encoding. And lo and behold I got his: rse@Alibaba:~/test$ cat type ANSI_X3.4-1968 rse@Alibaba:~/test$ So, here the getpreferredencoding() function returns ANSI_X3.4-1968 instead of UTF-8 !? 6) The solution is then obvious: open the file by specifying the encoding; a suggestion made already by Chris A. 7) Now, that source of the problem is known, I must investigate why my run-time environment differs from the interpreter environment. I know it is the same machine, same Python 3.4.0. As the mod_wsgi module in Apache2 initiates Python for the run-time, I will look there around. Dear All, Thank you each and everyone for your contribution. I would suggest to close this subject. If I get a solution around mod_wsgi + python I will post it. Kind regards to all, René -- https://mail.python.org/mailman/listinfo/python-list
Re: file.write() of non-ASCII characters differs in Interpreted Python than in script run
On Wednesday, August 26, 2015 at 5:59:12 PM UTC+2, Chris Angelico wrote:
> On Thu, Aug 27, 2015 at 1:02 AM, RAH wrote:
> > 7) Now, that source of the problem is known, I must investigate why my
> > run-time environment differs from the interpreter environment. I know it is
> > the same machine, same Python 3.4.0. As the mod_wsgi module in Apache2
> > initiates Python for the run-time, I will look there around.
> >
> First place to look would be the environment. If os.environ["LANG"]
> has "C" when you run under Apache, that would be the explanation. But
> explicitly choosing the encoding is the best way for other reasons
> anyway, and it solves the problem, so researching this is merely a
> matter of curiosity.
>
> ChrisA
Hello Chris,
Thanks for your further input.
os.environ{"LANG"} returns `en_US.UTF-8`, exactly the same as asking in Bash
`echo $LANG`. But again this is the interpreter.
Now if I ask the same os.environ["LANG"] within my application, it returns `C`
So, again, there is a marked difference between what the interpreter shows and
what the run-time shows.
In the meantime, I have checked the configuration directives of mod_wsgi. There
is nothing there to set or choose a particular environment. And I wonder if
there is a reason to check the Apache2 directives, like SetEnv ? Indeed Apache2
doesn't know anything about Python.
On the other hand the server I use (Ubuntu) still has Python 2.7 installed and
I can't remove it with apt-get. I believe Ubuntu needs python a lot. The Python
3.4 has been installed separately and it could be that a doubtful configuration
subsists.
Nevertheless I can't figure out why calling Python in the shell (interactive
mode) or letting mod_wsgi start the same Python provide two different
environments. I guess I must investigate that part because mod_wsgi gets to
Python in what I would call 'auto-discovery' mode. Obviously it gets the same
version 3.4.0 but maybe it picks up some 2.7 configuration files because the
installation of 3.4 next to 2.7 might not be perfect. I'll look at it.
Thank you Chris. If I find something I'll post it here.
René
--
https://mail.python.org/mailman/listinfo/python-list
Re: file.write() of non-ASCII characters differs in Interpreted Python than in script run
Dear All,
The solution / explanation follows.
Thanks to Graham Dumpleton, the author of mod_wsgi (the WSGI module for
Apache2) the source of the problem could be traced back to variables in
Apache2. Below are the details reproduced from
https://groups.google.com/forum/#!topic/modwsgi/4wdfCOnMUkU
Now, everything is indeed UTF-8 !
Thanks again to each and everyone.
Best regards,
René
Solution
There is a file /etc/apache2/envvars referred to by /etc/apache2/apache2.conf.
In that file, I found the following lines:
## The locale used by some modules like mod_dav
export LANG=C
## Uncomment the following line to use the system default locale instead:
#. /etc/default/locale
As I don't need mod_dav, neither is it compiled with Apache2 ($apache2ctl -l),
neither is it loaded with Apache2 ($apache2ctl -M), I commented / uncommented
the 2 lines so that it now looks like:
#export LANG=C
. /etc/default/locale
export LANG
After a stop/start of Apache2, everything works fine and when I put the code:
from locale import getpreferredencoding
prefcoding = getpreferredencoding()
from os import environ
lang = environ["LANG"]
g = open('envresults', 'a')
g.write('LANG: ' + lang + '\n')
g.write('PrefCod: ' + prefcoding + '\n')
in my WSGI application, it gives me the same as the interpreter:
rse@Alibaba:~/test$ cat envresults
LANG: en_US.UTF-8
PrefCod: UTF-8
rse@Alibaba:~/test$
-*- The End -*-
--
https://mail.python.org/mailman/listinfo/python-list
Loop Iteration
Hello.
I am writing a script to parse my logfiles to monitor sshd attempted
logins. I know I'm reinventing the wheel, but it's something I want
to do. The problem I am having is that upon the 4th or 5th pass in my
for statement I recieve an error
AttributeError: 'NoneType' object has no
attribute 'group'When I tested against a smaller
version of my logs that contained roughly 8 ip's (one ip appears 6
times another 2) it works fine without the AttributeError.
My question is if it is in my implementation of RE or could it be a
memory issue?
This is a sample of what the log looks like [code:1:eaa2962442]
Dec 25 11:30:17 linux sshd[2198]: Received
disconnect from :::200.91.12.4: 11: Bye Bye
Dec 25 11:30:18 linux sshd[2199]:
input_userauth_request: illegal user sibylla
Dec 25 11:30:18 linux sshd[2199]: Could not
reverse map address 200.91.12.4.
Dec 25 11:30:18 linux sshd[2199]: Failed password
for illegal user sibylla from :::200.91.12.4 port
55697 ssh2[/code:1:eaa2962442]
Actual script[code:1:eaa2962442]
import re, string
def main():
match = 'Failed password for illegal user'
pattern = re.compile(match)
f = open('xaf', 'r')
instance = 0
for line in f:
if pattern.findall(line):
this = re.sub(
r'^([a-zA-Z]+)\s*([0-9]+)\s*([0-9]+):([0-9]+):([0-9]+)\s*([a-z]+)\s*([a-z]+)\s*([^0-9]+)\s*([0-9]+)\s*([^0-9]+)',
'', line, 1)
ip =
re.match(r'^(?P([0-9]+).([0-9]+).([0-9]+).([0-9]))',
this)
of = open("out.txt", 'a')
print ip.group('ip')
instance = instance + 1
of.close()
f.close()
if instance != 0:
print "%s match(s) found for Failed password for
illegal user" % instance
if __name__ == "__main__":
main()
[/code:1:eaa2962442]
--
http://mail.python.org/mailman/listinfo/python-list
