Re: Unicode list
> When printing a list, the individual elements are converted with repr(), > not with str(). For a string object, repr() adds escape codes for all > bytes that are not printable ASCII characters. Thanks Martin, you're right, it were the repr() calls that messed up the output. Iterating the array like you proposed is even 1/100s faster ;) Regards, Rehceb -- http://mail.python.org/mailman/listinfo/python-list
Re: Unicode list
Rehceb Rotkiv schrieb: > Hello, > > I have this little grep-like program: > > ++snip++ > #!/usr/bin/python > > import sys > import re > > pattern = sys.argv[1] > inputfile = file(sys.argv[2], 'r') > > for line in inputfile: > matches = re.findall(pattern, line) > if matches: > print matches > ++snip++ > > Like this, the program prints some characters as strange escape > sequences, which is due to the input file being encoded in utf-8 As Paul said, your terminal is likely set to iso-8859 encoding, which is why it doesn't display UTF-8 correctly. The above program produces correct UTF-8 output. What you could do is: 1. read the file in as unicode 2. print the unicode to the terminal (will use the terminal encoding) or convert the unicode to strings with an explicit encoding before printing codecs.open() is very helpful for step 1, BTW. Georg -- http://mail.python.org/mailman/listinfo/python-list
Re: Unicode list
> Like this, the program prints some characters as strange escape > sequences, which is due to the input file being encoded in utf-8: When I > convert "re.findall..." to a string and wrap an "unicode()" around it, > the matches get printed correctly. Is it possible to make "matches" > unicode without saving it as a single string first? The function "unicode > ()" seems only to work for strings. Or is there a general way of telling > Python to abandon the ancient and evil land of iso-8859 for good and use > utf-8 only? Python does not live in the ancient and evi land of iso-8859; it lives in the ancient and evil land of ASCII. When printing a list, the individual elements are converted with repr(), not with str(). For a string object, repr() adds escape codes for all bytes that are not printable ASCII characters. To avoid this call to repr, you need to iterate over the list yourself, and print it: if matches: for m in matches: print m, print HTH, Martin -- http://mail.python.org/mailman/listinfo/python-list
Re: Unicode list
Rehceb Rotkiv wrote: > Hello, > > I have this little grep-like program: > > ++snip++ > #!/usr/bin/python > > import sys > import re > > pattern = sys.argv[1] > inputfile = file(sys.argv[2], 'r') > > for line in inputfile: > matches = re.findall(pattern, line) > if matches: > print matches > ++snip++ > > Like this, the program prints some characters as strange escape > sequences, which is due to the input file being encoded in utf-8: So the UTF-8 data gets printed to your terminal which isn't configured for UTF-8, right? > When I convert "re.findall..." to a string and wrap an "unicode()" around it, > the matches get printed correctly. How do you meaningfully convert it to a string? The matches variable refers to a list, but you surely don't want to be dealing with the list's string representation. > Is it possible to make "matches" unicode without saving it as a single string > first? Why not convert your input into Unicode and then, for the benefit of certain kinds of character classes, use re.findall in Unicode mode (by specifying re.U as a flag)? Then, each match will be produced as a Unicode object. > The function "unicode()" seems only to work for strings. Or is there a > general way of telling > Python to abandon the ancient and evil land of iso-8859 for good and use > utf-8 only? The only refuge from ancient and evil lands is found by climbing the mountain of Unicode: convert from encoded text as soon as you can, work only with Unicode objects, produce encoded text only when necessary. Paul -- http://mail.python.org/mailman/listinfo/python-list
Unicode list
Hello, I have this little grep-like program: ++snip++ #!/usr/bin/python import sys import re pattern = sys.argv[1] inputfile = file(sys.argv[2], 'r') for line in inputfile: matches = re.findall(pattern, line) if matches: print matches ++snip++ Like this, the program prints some characters as strange escape sequences, which is due to the input file being encoded in utf-8: When I convert "re.findall..." to a string and wrap an "unicode()" around it, the matches get printed correctly. Is it possible to make "matches" unicode without saving it as a single string first? The function "unicode ()" seems only to work for strings. Or is there a general way of telling Python to abandon the ancient and evil land of iso-8859 for good and use utf-8 only? Regards, Rehceb -- http://mail.python.org/mailman/listinfo/python-list
