On Feb 2, 2012, at 2:24 PM, peter wrote:
I am reading some text from a web site, using f=urllib.urlopen(),
and then s=f.read()
I then extract a bit of 's' as s1, s1 contains Na Ponta Do Pé
The é is encoded in a single byte as 0XE9.
If I do IS_SLUG.urlify(s1) it throws and error because 0XE9 is not a
valid character. I believe the encoding is ansii. I have tried all
manner of encoding and decoding but cannot get anything to work. If I
print s1 to the console or a file, then it works fine. But most python
character operations fail, presumably because they are expecting utf-8
which encodes é as two bytes.
If I do
s1=Na Ponta Do Pé
IS_SLUG.urlify(s1)
There is no error.
Clearly I could check for 0XE9 and convert it uniquely, but I wonder
if anyone could suggest a conversion that would work for any ansii
character. I have googled and experimented a lot on this with no
success.
The page you're reading is encoded as Latin-1. You need to decode it first.
