first of all, the stack trace suggests you have not set the "encoding" parameter on create_engine() as it's still using UTF-8.
If you mean that a single database column may have different encodings in different rows, you want to do your own encoding/decoding with "encoding errors" set to something liberal like "ignore". You also need to use your own custom type, as below: from sqlalchemy import types class MyEncodedType(types.TypeDecorator): impl = String def process_bind_param(self, value, dialect): assert isinstance(value, unicode) return value.encode('latin-1') def process_result_value(self, value, dialect): return value.decode('latin-1', 'ignore') then use MyEncodedType() as the type for all your columns which contain random encoding. No convert_unicode setting should be used on your engine as this type replaces that usage. On Jun 26, 2008, at 6:55 PM, Hermann Himmelbauer wrote: > > Hi, > I'm trying to access a database via SA, which contains varchars with > different, arbitrary encodings. Most of them are ascii or ISO-8859-2 > encoded, > however, many are windows-1252 encoded and there are also some other > weird > ones. > > In my engine setup, I set the encoding to latin1 and set > convert_unicode to > True, as I my application requires the database values in unicode > format. > > If SA now tries to retrieve such a key, the following traceback > occurs: > > ------------------ > File "/home/dusty/prog/python_modules/sqlalchemy/engine/base.py", > line 1605, > in _get_col > return processor(row[index]) > File "/home/dusty/prog/python_modules/sqlalchemy/databases/ > maxdb.py", line > 112, in process > return value.decode(dialect.encoding) > > File "/local/home/dusty/python/Python-2.4.4/lib/python2.4/encodings/ > utf_8.py", > line 16, in decode > return codecs.utf_8_decode(input, errors, True) > UnicodeDecodeError: 'utf8' codec can't decode bytes in position 3-6: > invalid > data > ----------------- > > What can I do? It's not so important that all characters are correctly > displayed, but it's vital that such improper encodings do not crash my > application. Perhaps, there's some "universal" encoding that is able > to deal > with such problems? > > Best Regards, > Hermann > > -- > [EMAIL PROTECTED] > GPG key ID: 299893C7 (on keyservers) > FP: 0124 2584 8809 EF2A DBF9 4902 64B4 D16B 2998 93C7 > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~----------~----~----~----~------~----~------~--~---