I could be wrong, but I'm fairly sure that \xc3\xa3 is a 4 byte unicode sequence (which is essentially UTF-16).
I'm not hot on understanding how unicode works, but I've had to deal with it hundreds of times due to the nature of the webapps our company writes (most data mining and indexing adapters). You have a couple of options. * Strip out all non convertible characters (errors=strip or errors=ignore can't remember which) * Convert all non convertible characters to the "?" sign (errors=replace) * Use to_unicode_or_bust() - see http://pastebin.com/qrFk2zSp - not perfect but does the job. (you'll also see a clean() in there which does stripping of all 4 byte sequences). For a better understanding of why this happens, I recommend looking at the following: https://github.com/kumar303/unicode-in-python Something I've learned to accept in Python 2.*, is the complete and utter lack of "appropriate" unicode handling. By this, I mean that tasks which should be relatively simple, are not. We ended up making a String() object which did all these appropriate sanity checks (based on the pastebin link above) and we have never had problems since. Hope this helps Cal Cal On Sun, Jun 12, 2011 at 11:31 PM, alexandre...@gmail.com < alexandre...@gmail.com> wrote: > Hi, > > I just converted a mssql database to sqlite3. > I used the folloing text_factory > > con.text_factory = lambda x: x.decode('iso-8859-1').encode('utf-8') > > that just converts from latin1 to utf-8 > where i browse the tables in python shell it seems everything is fine > and well converted > but when in admin, I just get the error > "could not decode utf-8 column" > > the columns django complains has the following data '4760 Vila Nova de > Famalic\xc3\xa3o ' > > Any help appreciated. > > -- > You received this message because you are subscribed to the Google Groups > "Django users" group. > To post to this group, send email to django-users@googlegroups.com. > To unsubscribe from this group, send email to > django-users+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/django-users?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com. To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
