On Feb 18, 2014, at 5:42 PM, Valentino Volonghi <dialt...@adroll.com> wrote:
> Hey guys, > > we've moved recently to SQLAlchemy 0.9.2 from 0.7.9. > > This move coincided with the introduction of UnicodeEncoreErrors in our > system. They appear to be happening at random and have no real way for us to > debug as we can't really reproduce them, except that they happen in our > system and the tracebacks lead directly to the insides of sqlalchemy. > > https://gist.github.com/dialtone/9081835 > > This is the traceback we get, with the nice and clear: > UnicodeEncodeError: 'ascii' codec can't encode character u'\xdc' in position > 1: ordinal not in range(128) > > Our PG 9.3 is setup with encoding at utf8, we also have the client_encoding > set at utf8 but it still seems that the library randomly picks what to do in > that spot. > > The stacktrace points to this error: > https://github.com/zzzeek/sqlalchemy/blob/rel_0_9_2/lib/sqlalchemy/orm/strategies.py#L154 > > Almost no matter what our calls are, when we fetch an object with a Unicode > field that is actually using multi-bytes it ends up failing point to that > line with the UnicodeEncodeError. > > If I were to trust my guts I'd say it might be related to py3k support but > I'd probably be wrong. > > Can anyone help us figure out what this issue might be? OK some questions: 1. what options are you passing to create_engine()? 2. what kind of column types and data do you have in the table that is mapped to the “users” mapping denoted by: self.organization.users ? 3. are you running with the C extensions installed? what happens differently if you run without the C extensions? (depends on how reproducible this is) the stack trace doesn’t say much here as it seems like C code is where things are going wrong. 4. is this error only under load ? it may be the issue noted by Jeff in another email, or not. 5. Potential workaround, assuming the issue is the one Jeff refers to. Do this in your application startup, *before* any requests are served: # create engine normally eng = create_engine(…) # initialize the dialect. conn = eng.connect() conn.close() specifically, do the above anytime you are calling create_engine(). Note that the app would ideally be calling create_engine() only once per URL. If your app is using an anti pattern like “create_engine per operation” or something like that, that could cause more problems (even it not, it is very non-performant). 6. more extreme workaround, if Amazon RDS is being weird here: # create engine normally eng = create_engine(…) # initialize the dialect. conn = eng.connect() conn.close() # override the unicode returns detection to force SQLAlchemy to check for # decode in all cases (will perform poorly if no C extensions are in use or pre-0.9.2 in use) eng.dialect.returns_unicode_strings = “conditional" let me know how it goes, thanks!
signature.asc
Description: Message signed with OpenPGP using GPGMail
