[sqlalchemy] compiles() no longer working with PrimaryKeyConstraint after update from 0.8 to 0.9.2
Hi guys,
After updating to 0.9.2 from 0.8 my custom compiles() are not working any
longer. The definition looks likes this:
@compiles(PrimaryKeyConstraint, postgresql)def
add_pg_primary_key_options(constraint, compiler, **kwargs):
ddl = compiler.visit_primary_key_constraint(constraint, **kwargs)
if 'postgresql_fillfactor' in constraint.kwargs:
fillfactor = constraint.kwargs.get('postgresql_fillfactor')
pos = ddl.index(')') + 1
ddl = ddl[:pos] + WITH (FILLFACTOR=%s) % fillfactor + ddl[pos:]
return ddl
But now I get this error:
sqlalchemy.exc.ArgumentError: Argument 'postgresql_fillfactor' is not
accepted by dialect 'postgresql' on behalf of class
'sqlalchemy.sql.schema.PrimaryKeyConstraint'
It seems the function itself is not executed at all. Are there any changes
in 0.9+ I am not aware of that changed the compiles() behaviour?
Thanks,
Adrian
--
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/groups/opt_out.
Re: [sqlalchemy] compiles() no longer working with PrimaryKeyConstraint after update from 0.8 to 0.9.2
On Feb 18, 2014, at 4:59 AM, Adrian adrian.schre...@gmail.com wrote:
Hi guys,
After updating to 0.9.2 from 0.8 my custom compiles() are not working any
longer. The definition looks likes this:
@compiles(PrimaryKeyConstraint, postgresql)
def add_pg_primary_key_options(constraint, compiler, **kwargs):
ddl = compiler.visit_primary_key_constraint(constraint, **kwargs)
if 'postgresql_fillfactor' in constraint.kwargs:
fillfactor = constraint.kwargs.get('postgresql_fillfactor')
pos = ddl.index(')') + 1
ddl = ddl[:pos] + WITH (FILLFACTOR=%s) % fillfactor + ddl[pos:]
return ddl
But now I get this error:
sqlalchemy.exc.ArgumentError: Argument 'postgresql_fillfactor' is not
accepted by dialect 'postgresql' on behalf of class
'sqlalchemy.sql.schema.PrimaryKeyConstraint'
It seems the function itself is not executed at all. Are there any changes in
0.9+ I am not aware of that changed the compiles() behavior?
your function is fine but the issue is that PrimaryKeyConstraint here is
checking on that argument and raising right in the constructor, as we now check
on those arguments (see
http://docs.sqlalchemy.org/en/latest/changelog/changelog_09.html#change-2df4f7fe29c0f5aa2f957f4a89b0d74d
).
two options here are to add it in as an argument:
from sqlalchemy.dialects.postgresql import base
base.PGDialect.construct_arguments.append((PrimaryKeyConstraint, {fillfactor:
None}))
pk = PrimaryKeyConstraint('a', info=dict(postgresql_fillfactor='x'))
note the above should be considered a workaround; I should add an API feature
to support adding new arguments to dialects here. for now the above call has
to be before any schema objects are constructed or it won’t take effect.
the other option is to use .info instead (the inconvenience here is that ‘info’
isn’t accepted in the constructor…):
@compiles(PrimaryKeyConstraint, postgresql)
def add_pg_primary_key_options(constraint, compiler, **kwargs):
ddl = compiler.visit_primary_key_constraint(constraint, **kwargs)
if 'postgresql_fillfactor' in constraint.info:
fillfactor = constraint.info.get('postgresql_fillfactor')
pos = ddl.index(')') + 1
ddl = ddl[:pos] + WITH (FILLFACTOR=%s) % fillfactor + ddl[pos:]
return ddl
pk = PrimaryKeyConstraint('a')
pk.info['postgresql_fillfactor'] = 'x'
Thanks,
Adrian
--
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To unsubscribe from this group and stop receiving emails from it, send an
email to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/groups/opt_out.
signature.asc
Description: Message signed with OpenPGP using GPGMail
[sqlalchemy] Issue with cx_oracle
The docstring for the cx-Oracle dialog says:
SQLAlchemy will pass all unicode strings directly to cx_oracle, and
additionally uses an output handler so that all string based result
values are returned as unicode as well.
The latter does no longer seem to be true; the handler was recently
removed with ticket 2911.
So now when I have varchar2 columns and do a simple query like this one,
I get encoded strings instead of unicode as before (in Python 2):
engine = create_engine('oracle+cx_oracle://..')
con = engine.connect()
for row in con.execute(select username from users):
print row
Is this really intended? What am I supposed to do when I want to always
get unicode back?
-- Christoph
--
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/groups/opt_out.
Re: [sqlalchemy] Issue with cx_oracle
On Feb 18, 2014, at 2:06 PM, Christoph Zwerschke c...@online.de wrote:
The docstring for the cx-Oracle dialog says:
SQLAlchemy will pass all unicode strings directly to cx_oracle, and
additionally uses an output handler so that all string based result values
are returned as unicode as well.
The latter does no longer seem to be true; the handler was recently removed
with ticket 2911.
good catch, I’ve rewritten the docs here:
http://docs.sqlalchemy.org/en/latest/dialects/oracle.html#unicode. The more
common approach of using text() is included.
So now when I have varchar2 columns and do a simple query like this one, I
get encoded strings instead of unicode as before (in Python 2):
engine = create_engine('oracle+cx_oracle://..')
con = engine.connect()
for row in con.execute(select username from users):
print row
Is this really intended? What am I supposed to do when I want to always get
unicode back?
it is very unfortunate but cx_Oracle has an unacceptable performance penalty
for using this feature. As you can see here:
https://github.com/pydata/pandas/issues/2717#issuecomment-29046644 if I hadn’t
been alerted to this, users who didn’t have the courtesy to notify me of this
problem were ready to have the Pandas project entirely dump consideration of
SQLAlchemy integration, for its supposed “cruft” and 2-4x the CPU cycles on
top of your database driver”, when in fact this cruft and overhead is entirely
within cx_Oracle. All due to two lines of code.
Needless to say I’m a bit miffed that cx_Oracle’s huge performance bug doesn’t
impact the reputation of cx_Oracle, but instead harms and slanders the
SQLAlchemy project. I’ve not had good results alerting cx_Oracle to other
issues in the past (issues related to the RETURNING feature, two-phase
transactions, etc) so it’s not really worth trying to get traction on this one.
So until a solution is found to the outputtypehandler unicode issue in
cx_Oracle Python 2, the recipe in the new docs are how it has to be for now
(which is similar for other backends anyway), that is:
from sqlalchemy import text, Unicode
result = conn.execute(text(select username from
user).columns(username=Unicode))
As far as restoring the outputtypehandler, i didn’t have good results trying to
come up with a recipe to add one in from the outside that “nests” the one we
already apply there for decimals. So it you really need this as it was, using
outputtypehandler, I will accept a pullreq that restores it, but turned off by
default. A flag called “coerce_to_unicode”, analogous to “coerce_to_decimal”,
will default to False but when set to True will use the unicode
outputtypehandler.
signature.asc
Description: Message signed with OpenPGP using GPGMail
[sqlalchemy] SQLAlchemy 0.9.2 UnicodeEncodeErrors with PG 9.3.2 on Amazon RDS
Hey guys, we've moved recently to SQLAlchemy 0.9.2 from 0.7.9. This move coincided with the introduction of UnicodeEncoreErrors in our system. They appear to be happening at random and have no real way for us to debug as we can't really reproduce them, except that they happen in our system and the tracebacks lead directly to the insides of sqlalchemy. https://gist.github.com/dialtone/9081835 This is the traceback we get, with the nice and clear: UnicodeEncodeError: 'ascii' codec can't encode character u'\xdc' in position 1: ordinal not in range(128) Our PG 9.3 is setup with encoding at utf8, we also have the client_encoding set at utf8 but it still seems that the library randomly picks what to do in that spot. The stacktrace points to this error: https://github.com/zzzeek/sqlalchemy/blob/rel_0_9_2/lib/sqlalchemy/orm/strategies.py#L154 Almost no matter what our calls are, when we fetch an object with a Unicode field that is actually using multi-bytes it ends up failing point to that line with the UnicodeEncodeError. If I were to trust my guts I'd say it might be related to py3k support but I'd probably be wrong. Can anyone help us figure out what this issue might be? Thanks! -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/groups/opt_out.
[sqlalchemy] Re: SQLAlchemy 0.9.2 UnicodeEncodeErrors with PG 9.3.2 on Amazon RDS
On Tuesday, February 18, 2014 2:42:36 PM UTC-8, Valentino Volonghi wrote: Can anyone help us figure out what this issue might be? One last thing here is that our driver is psycopg 2.5.2 -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/groups/opt_out.
Re: [sqlalchemy] SQLAlchemy 0.9.2 UnicodeEncodeErrors with PG 9.3.2 on Amazon RDS
On Tue, Feb 18, 2014 at 02:42:36PM -0800, Valentino Volonghi wrote: Hey guys, we've moved recently to SQLAlchemy 0.9.2 from 0.7.9. This move coincided with the introduction of UnicodeEncoreErrors in our system. They appear to be happening at random and have no real way for us to debug as we can't really reproduce them, except that they happen in our system and the tracebacks lead directly to the insides of sqlalchemy. https://gist.github.com/dialtone/9081835 This is the traceback we get, with the nice and clear: UnicodeEncodeError: 'ascii' codec can't encode character u'\xdc' in position 1: ordinal not in range(128) Our PG 9.3 is setup with encoding at utf8, we also have the client_encoding set at utf8 but it still seems that the library randomly picks what to do in that spot. The stacktrace points to this error: https://github.com/zzzeek/sqlalchemy/blob/rel_0_9_2/lib/sqlalchemy/orm/strategies.py#L154 Almost no matter what our calls are, when we fetch an object with a Unicode field that is actually using multi-bytes it ends up failing point to that line with the UnicodeEncodeError. If I were to trust my guts I'd say it might be related to py3k support but I'd probably be wrong. Can anyone help us figure out what this issue might be? I've been having this same (I think) problem for some time. Running sqlalchemy 0.9.2, using the mysqldb driver, python 2.6. The problem has been happening sporadically on our production server. Until today, I could not reproduce it in a test environment, so I've slowly been adding debug logging to the production code to try to see what's going on. I've finally figured it out, I think, just a couple of hours ago, as it turns out. There is a race condition having to do with calling dialect.initialize(). If multiple threads are clamoring to access the database when the app starts, some of them may get to the database before the dialect is completely initialized. Since dialect.initialize() is responsible for (among other things) correctly setting dialect.returns_unicode_strings, this can result in sqlalchemy trying to erroneously attempting to decoding unicode strings to unicode (which results in the UnicodeEncodeError.) Anyhow, bug report is at: https://bitbucket.org/zzzeek/sqlalchemy/issue/2964/ As a workaround, at app config time, right after create_engine is called, I execute a query (before there is a possibility of a multi-thread race.) E.g. engine = sa.create_engine(...) # early query to force dialect.initialize() engine.execute(sa.sql.select([1])) Jeff -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/groups/opt_out.
Re: [sqlalchemy] SQLAlchemy 0.9.2 UnicodeEncodeErrors with PG 9.3.2 on Amazon RDS
On Feb 18, 2014, at 5:42 PM, Valentino Volonghi dialt...@adroll.com wrote: Hey guys, we've moved recently to SQLAlchemy 0.9.2 from 0.7.9. This move coincided with the introduction of UnicodeEncoreErrors in our system. They appear to be happening at random and have no real way for us to debug as we can't really reproduce them, except that they happen in our system and the tracebacks lead directly to the insides of sqlalchemy. https://gist.github.com/dialtone/9081835 This is the traceback we get, with the nice and clear: UnicodeEncodeError: 'ascii' codec can't encode character u'\xdc' in position 1: ordinal not in range(128) Our PG 9.3 is setup with encoding at utf8, we also have the client_encoding set at utf8 but it still seems that the library randomly picks what to do in that spot. The stacktrace points to this error: https://github.com/zzzeek/sqlalchemy/blob/rel_0_9_2/lib/sqlalchemy/orm/strategies.py#L154 Almost no matter what our calls are, when we fetch an object with a Unicode field that is actually using multi-bytes it ends up failing point to that line with the UnicodeEncodeError. If I were to trust my guts I'd say it might be related to py3k support but I'd probably be wrong. Can anyone help us figure out what this issue might be? OK some questions: 1. what options are you passing to create_engine()? 2. what kind of column types and data do you have in the table that is mapped to the “users” mapping denoted by: self.organization.users ? 3. are you running with the C extensions installed? what happens differently if you run without the C extensions? (depends on how reproducible this is) the stack trace doesn’t say much here as it seems like C code is where things are going wrong. 4. is this error only under load ? it may be the issue noted by Jeff in another email, or not. 5. Potential workaround, assuming the issue is the one Jeff refers to. Do this in your application startup, *before* any requests are served: # create engine normally eng = create_engine(…) # initialize the dialect. conn = eng.connect() conn.close() specifically, do the above anytime you are calling create_engine(). Note that the app would ideally be calling create_engine() only once per URL. If your app is using an anti pattern like “create_engine per operation” or something like that, that could cause more problems (even it not, it is very non-performant). 6. more extreme workaround, if Amazon RDS is being weird here: # create engine normally eng = create_engine(…) # initialize the dialect. conn = eng.connect() conn.close() # override the unicode returns detection to force SQLAlchemy to check for # decode in all cases (will perform poorly if no C extensions are in use or pre-0.9.2 in use) eng.dialect.returns_unicode_strings = “conditional let me know how it goes, thanks! signature.asc Description: Message signed with OpenPGP using GPGMail
[sqlalchemy] Re: SQLAlchemy 0.9.2 UnicodeEncodeErrors with PG 9.3.2 on Amazon RDS
I've had problems with this in the past. I thought it was SqlAlchemy, and it ended up being issues with my data in Pyramid (the Pylons successor). A few items had the wrong character encoding -- I was expecting it to be UTF8/ASCII, but it was using a different charset. It could be the issue that Jeff mentioned, but I'd suggest writing some exception handlers to log where this is happening in Pylons and try to dump all the relevant data. I have a custom exception logger in Pyramid that logs these types of errors into the database -- along with the url misc args; sometimes I'll drop some hints into the request object too -- like stashing a URL that is being processed or the integer ID of an object being worked on. then i can usually figure out where the problems are. I just want to bring this up, because the Pylons/Pyramid error reporting -- and the SqlAlchemy reporting -- can often make an error look to be in a different spot than it really is Sidenote: How are you liking the pg RDS support? -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/groups/opt_out.
