Hmm, this one has me stumped. As best I can tell after poking at it using the column_reflect event, a custom dialect, etc. - the issue here is that in pysqlite.py we (in my Python 3.3 install) are selecting `sqlite3.dbapi2` as the dbapi interface, but we aren't telling sqlite3 anything about how to treat unicode errors. From what I am reading (but it seems inconsistent, maybe?) sqlite3 automatically decodes all database retrieved values from their bytes for text fields, returning unicode strings. Except... that doesn't always seem to be true. I hex-edited a db file to change the utf-8 string "hello" to "hell" + 0x92 and sqlite3 switched from returning "hello" to b"hell\x92", or something like that - I've been poking at this for so long I've lost track of that transcript.
One can override sqlite3's text factory, apparently, with (for instance) `sqlite3.text_factory = lambda x: x.decode('utf-8', errors='ignore')`. Maybe the key is to try and find a way to trigger that from sqlalchemy? I tried and failed, maybe someone else can point me back to the path? Just to re-summarize the problem: In python 3, I'm getting errors trying to read a row from a sqlite database that has a TEXT column with an invalid utf-8 sequence (specifically, the singleton bye '0x92'). I'd love to just have sqlalchemy move along and ignore the byte, but I'm not clear how to do that. On Tuesday, February 4, 2014 4:33:46 AM UTC-8, Simon King wrote: > > I've not done much with reflection, but perhaps you could use the > column_reflect event: <snip> -- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/groups/opt_out.