On Oct 06, Vitaly Pashkov <ad...@fluda.net> wrote:
> You can get my copy here: http://fluda.net/personal/movies.list.gz
It's identical to the one I have.
> imdb=> select id, title from title order by random() limit 5;
> id | title
> ---------+-----------------------------------------
> 1147146 | (#3.11)
> 1010589 | (2007-05-21)
The above ones are not symptoms of problems: many episodes of
tv series are identified only by their #SEASON.EPISODE number
or by their airing date (when a title is missing, obviously).
> Got it!
[...]
> Auf der grünen Wiese
Not exactly the nasty title I expected - whatever it means. :-)
There's nothing wrong with it, and there are many more umlauts
in that list - this is probably the first occurrence that creates
problems when the data are flushed to the database (the data are
temporarily stored in a Python dictionary, and so they are not
flushed to the db in order).
> /usr/lib/python2.5/site-packages/IMDbPY-4.2-py2.5.egg/EGG-INFO/scripts/imdbpy2sql.py:628:
> UnicodeWarning: Unicode equal comparison failed to convert both
> arguments to Unicode - interpreting them as being unequal
> if ts[-1].lower() in _articles:
Try this simple script, to see if it creates any problem:
================================================================
#!/usr/bin/env python
import imdb
utf8_title = 'Auf der gr\xc3\xbcnen Wiese'
print utf8_title in imdb.utils._articles
================================================================
If it replies "False" without raising warnings of exceptions,
I think the problem is not in Python itself but in the psycopg2
module or in the configuration of PostgreSQL.
> BUT! I also tried to run it without my modification at line 1427 and with
> modified title_soundex. It stopped at the other line:
[...]
> "/usr/lib/python2.5/site-packages/IMDbPY-4.2-py2.5.egg/EGG-INFO/scripts/imdbpy2sql.py",
> line 1030, in _runCommand
> CURS.executemany(self.sqlstr, self.converter(dataList))
> psycopg2.DataError: byte sequence invalid for encoding "UTF8": 0xc333
> HINT: This error can also happen if the byte sequence does not match the
> encoding expected by the server, which is controlled by "client_encoding".
It's more or less expected: the data is processed without glitches,
but everything explodes when CURS.executemany tries to dump it into
the database.
To summarize how imdbpy2sql.py works: it reads the plain text data
files (which are mostly in iso-8859-1 encoding), convert them to
utf-8 for internal usage (for a series of more or less good reasons)
and uses a cursor provided by the db access module (psycopg2, in this
case) to store them (again, passing the strings as utf-8).
Your change forces imdbpy2sql.py to use _unicode_ representation of
titles; the UnicodeWarning you get is because it compares a unicode
string to a list of utf-8 encoded strings (imdb.utils._articles).
As a temporary solution you can convert _articles to a list of unicodes,
but I can't consider this a real fix. Put this line somewhere at the
top of the script - hoping it will not break something else ;-) :
_articles = [x.decode('utf8') for x in _articles]
Why psycopg2 or your PostgreSQL don't play nicely with utf-8 strings
is beyond my imagination. :-)
Another small test (this _could_ exclude psycopg2, even if it can
always be a matter of how it's initialized by SQLObject/SQLAlchemy):
===========================================================
#!/usr/bin/env python
import psycopg2
utf8_title = 'Auf der gr\xc3\xbcnen Wiese'
connection = psycopg2.connect(database='imdb', user='UNAME', password='PWD')
curs = connection.cursor()
curs.execute("INSERT INTO title (title, kind_id) VALUES (%s, 1);",
(utf8_title,))
connection.commit()
===========================================================
In my installation, it works.
As you can see, debugging these strange interactions between
charsets/encodings and various modules, libraries and database
engines is a real pain. :-(
Thanks for your time!
--
Davide Alberani <davide.alber...@gmail.com> [GPG KeyID: 0x465BFD47]
http://erlug.linux.it/~da/
------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf
_______________________________________________
Imdbpy-help mailing list
Imdbpy-help@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/imdbpy-help
