On Tue, Feb 18, 2014 at 02:42:36PM -0800, Valentino Volonghi wrote: > Hey guys, > > we've moved recently to SQLAlchemy 0.9.2 from 0.7.9. > > This move coincided with the introduction of UnicodeEncoreErrors in our > system. They appear to be happening at random and have no real way for us > to debug as we can't really reproduce them, except that they happen in our > system and the tracebacks lead directly to the insides of sqlalchemy. > > https://gist.github.com/dialtone/9081835 > > This is the traceback we get, with the nice and clear: > UnicodeEncodeError: 'ascii' codec can't encode character u'\xdc' in > position 1: ordinal not in range(128) > > Our PG 9.3 is setup with encoding at utf8, we also have the client_encoding > set at utf8 but it still seems that the library randomly picks what to do > in that spot. > > The stacktrace points to this error: > https://github.com/zzzeek/sqlalchemy/blob/rel_0_9_2/lib/sqlalchemy/orm/strategies.py#L154 > > Almost no matter what our calls are, when we fetch an object with a Unicode > field that is actually using multi-bytes it ends up failing point to that > line with the UnicodeEncodeError. > > If I were to trust my guts I'd say it might be related to py3k support but > I'd probably be wrong. > > Can anyone help us figure out what this issue might be?
I've been having this same (I think) problem for some time. Running sqlalchemy 0.9.2, using the mysqldb driver, python 2.6. The problem has been happening sporadically on our production server. Until today, I could not reproduce it in a test environment, so I've slowly been adding debug logging to the production code to try to see what's going on. I've finally figured it out, I think, just a couple of hours ago, as it turns out. There is a race condition having to do with calling dialect.initialize(). If multiple threads are clamoring to access the database when the app starts, some of them may get to the database before the dialect is completely initialized. Since dialect.initialize() is responsible for (among other things) correctly setting dialect.returns_unicode_strings, this can result in sqlalchemy trying to erroneously attempting to decoding unicode strings to unicode (which results in the UnicodeEncodeError.) Anyhow, bug report is at: https://bitbucket.org/zzzeek/sqlalchemy/issue/2964/ As a workaround, at app config time, right after create_engine is called, I execute a query (before there is a possibility of a multi-thread race.) E.g. engine = sa.create_engine(...) # early query to force dialect.initialize() engine.execute(sa.sql.select([1])) Jeff -- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/groups/opt_out.
