On Oct 05, Vitaly Pashkov <ad...@fluda.net> wrote:
> Trying to find out where exactly my
> problem occurs i slightly modified "readMovieList()" function at line 1427
> of imdbpy2sql.py, so now it looks like:
> mid = CACHE_MID.addUnique(title.decode('utf-8'), yearData)
I'm not sure that with such a change you'll get a consistent
database. :-/
Anyway it's worth a try, at least to spot the problem.
Two things you can try, now:
1. put your movies.list.gz somewhere: with the one I've downloaded
yesterday I'm unable to reproduce the problem; I'd like to try yours.
2. with your change to line 1427 in place, modify also the title_soundex
function adding at the top these two lines:
print _(title)
sys.stdout.flush()
in your output you should see the "wrong" title right before
the UnicodeWarning warning.
> and then it continues to importing data. This error occurs only once.
It starts to sound a lot like garbage in the data...
> Any ideas how i can intercept this UnicodeWarning so i can see at what line of
> movie.list it happen? I tried to add next line to readMovieList():
> print "Title: " + title + ", counter: " + str(count)
> but the output is soooo big... tonns of lines, over9000.
Try modifying the title_soundex function as said above, and
then run your command appending this:
2&>1 | tee ~/OUTPUT.txt
After that, you can easily search for UnicodeWarning in the
~/OUTPUT.txt file.
> BTW, after that change (decode('utf-8')) DB content is looking good, but
> as it contains almost 1.5M records i can't be sure for 100%.
As said, doing so I'm not too sure that the database will store the
titles with the right encoding (every database seems to have a
different opinion about how to handle input in unicode and/or utf8
or other encodings...)
Thank you very much for you help!
--
Davide Alberani <davide.alber...@gmail.com> [GPG KeyID: 0x465BFD47]
http://erlug.linux.it/~da/
------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf
_______________________________________________
Imdbpy-help mailing list
Imdbpy-help@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/imdbpy-help
