Re: [sqlalchemy] utf8 error upon insert
upgrading pymysql did the trick. thank you.
On Thursday, June 9, 2016 at 12:02:29 PM UTC-4, Mike Bayer wrote:
>
>
> VARBINARY should not have a utf-8 encoding step at all. I can replace
> VARBINARY directly in my script and there's no problem; can you upgrade
> your pymysql? I seem to recall someone having this problem recently.
>
>
> Also please run the script below (with your database URL) and send the
> full output, as it will include things like the SQL MODE you're running
> and other things:
>
> from sqlalchemy import *
> from sqlalchemy.orm import *
> from sqlalchemy.ext.declarative import declarative_base
>
> Base = declarative_base()
>
>
> class A(Base):
> __tablename__ = 'a'
> id = Column(Integer, primary_key=True)
> data = Column(VARBINARY(255))
>
> e = create_engine("mysql+pymysql://scott:tiger@localhost/test", echo=True)
> Base.metadata.drop_all(e)
> Base.metadata.create_all(e)
>
> gcm_key =
> '\xfeE\x87\xe7\xc9\xe5\xec\xe0\x9c\xd6\x85\x11\xc7\xebd\xe3\x7f\xd9\xfel\xe6\x86"j\xbe=\xf4\xd7\x95\x99F\x8f'
>
>
>
> s = Session(e)
> s.add(A(data=gcm_key))
> s.commit()
>
>
>
> On 06/09/2016 11:16 AM, Ven Karri wrote:
> > The only difference is that I am using sqlalchemy.types.VARBINARY(256)
> > instead of String(255)
> >
> > On Thursday, June 9, 2016 at 10:47:32 AM UTC-4, Mike Bayer wrote:
> >
> > what charset is in your my.cnf and/or how are you connecting. Mine
> > only
> > produces a warning. Here is an MCVE (definition: I can actually run
> > it):
> >
> > from sqlalchemy import *
> > from sqlalchemy.orm import *
> > from sqlalchemy.ext.declarative import declarative_base
> >
> > Base = declarative_base()
> >
> >
> > class A(Base):
> > __tablename__ = 'a'
> > id = Column(Integer, primary_key=True)
> > data = Column(String(255))
> >
> > e = create_engine("mysql+pymysql://scott:tiger@localhost/test",
> > echo=True)
> > Base.metadata.create_all(e)
> >
> > gcm_key =
> >
> '\xfeE\x87\xe7\xc9\xe5\xec\xe0\x9c\xd6\x85\x11\xc7\xebd\xe3\x7f\xd9\xfel\xe6\x86"j\xbe=\xf4\xd7\x95\x99F\x8f'
>
>
> >
> >
> > s = Session(e)
> > s.add(A(data=gcm_key))
> > s.commit()
> >
> >
> >
> >
> > output:
> >
> > python test.py
> >
> /home/classic/.venv/lib/python2.7/site-packages/pymysql/cursors.py:146:
> > Warning: Table 'test' already exists
> > result = self._query(query)
> >
> /home/classic/.venv/lib/python2.7/site-packages/pymysql/cursors.py:146:
> > Warning: Incorrect string value: '\xFEE\x87\xE7\xC9\xE5...' for
> column
> > 'data' at row 1
> >result = self._query(query)
> >
> >
> > please run this script and modify it to show your error thanks!
> >
> >
> >
> >
> >
> > On 06/09/2016 10:42 AM, Ven Karri wrote:
> > > Any ideas?
> > >
> > > On Thursday, June 9, 2016 at 10:21:27 AM UTC-4, Ven Karri wrote:
> > >
> > > Using, python 2.7 using 'mysql+pymysql' driver
> > >
> > > Code is very simple:
> > >
> > > gcm_key =
> > >
> >
> '\xfeE\x87\xe7\xc9\xe5\xec\xe0\x9c\xd6\x85\x11\xc7\xebd\xe3\x7f\xd9\xfel\xe6\x86"j\xbe=\xf4\xd7\x95\x99F\x8f'
>
>
> >
> > > model = Sample(gcm_key=gcm_key)
> > > session.add(model)
> > >
> > > Now what you said is to make it u'some string'. The string in
> > > question here is the gcm_key. So, I did this:
> > >
> > > gcm_key = gcm_key.decode('utf8')
> > >
> > > That throws an error:
> > >
> > > UnicodeDecodeError: 'utf8' codec can't decode byte 0xfe in
> > position
> > > 0: invalid start byte
> > >
> > > On Thursday, June 9, 2016 at 10:04:49 AM UTC-4, Mike Bayer
> wrote:
> > >
> > >
> > >
> > > On 06/09/2016 09:52 AM, Ven Karri wrote:
> > > > I am getting a UTF-8 error upon insert using sql alchemy
> > ORM.
> > > The same
> > > > query runs fine when I run using raw sql. Here's the ORM
> &
Re: [sqlalchemy] utf8 error upon insert
The only difference is that I am using sqlalchemy.types.VARBINARY(256)
instead of String(255)
On Thursday, June 9, 2016 at 10:47:32 AM UTC-4, Mike Bayer wrote:
>
> what charset is in your my.cnf and/or how are you connecting. Mine only
> produces a warning. Here is an MCVE (definition: I can actually run it):
>
> from sqlalchemy import *
> from sqlalchemy.orm import *
> from sqlalchemy.ext.declarative import declarative_base
>
> Base = declarative_base()
>
>
> class A(Base):
> __tablename__ = 'a'
> id = Column(Integer, primary_key=True)
> data = Column(String(255))
>
> e = create_engine("mysql+pymysql://scott:tiger@localhost/test", echo=True)
> Base.metadata.create_all(e)
>
> gcm_key =
> '\xfeE\x87\xe7\xc9\xe5\xec\xe0\x9c\xd6\x85\x11\xc7\xebd\xe3\x7f\xd9\xfel\xe6\x86"j\xbe=\xf4\xd7\x95\x99F\x8f'
>
>
>
> s = Session(e)
> s.add(A(data=gcm_key))
> s.commit()
>
>
>
>
> output:
>
> python test.py
> /home/classic/.venv/lib/python2.7/site-packages/pymysql/cursors.py:146:
> Warning: Table 'test' already exists
>result = self._query(query)
> /home/classic/.venv/lib/python2.7/site-packages/pymysql/cursors.py:146:
> Warning: Incorrect string value: '\xFEE\x87\xE7\xC9\xE5...' for column
> 'data' at row 1
> result = self._query(query)
>
>
> please run this script and modify it to show your error thanks!
>
>
>
>
>
> On 06/09/2016 10:42 AM, Ven Karri wrote:
> > Any ideas?
> >
> > On Thursday, June 9, 2016 at 10:21:27 AM UTC-4, Ven Karri wrote:
> >
> > Using, python 2.7 using 'mysql+pymysql' driver
> >
> > Code is very simple:
> >
> > gcm_key =
> >
> '\xfeE\x87\xe7\xc9\xe5\xec\xe0\x9c\xd6\x85\x11\xc7\xebd\xe3\x7f\xd9\xfel\xe6\x86"j\xbe=\xf4\xd7\x95\x99F\x8f'
>
>
> > model = Sample(gcm_key=gcm_key)
> > session.add(model)
> >
> > Now what you said is to make it u'some string'. The string in
> > question here is the gcm_key. So, I did this:
> >
> > gcm_key = gcm_key.decode('utf8')
> >
> > That throws an error:
> >
> > UnicodeDecodeError: 'utf8' codec can't decode byte 0xfe in position
> > 0: invalid start byte
> >
> > On Thursday, June 9, 2016 at 10:04:49 AM UTC-4, Mike Bayer wrote:
> >
> >
> >
> > On 06/09/2016 09:52 AM, Ven Karri wrote:
> > > I am getting a UTF-8 error upon insert using sql alchemy ORM.
> > The same
> > > query runs fine when I run using raw sql. Here's the ORM
> query:
> > >
> > > rotating_keys_object = rotating_keys_model(
> > > gcm_key=rot_gcm_key,
> > > nonce=rot_nonce,
> > > tag=rot_tag,
> > > operational_team_id=self.operational_team_id
> > > )
> > > session.add(rotating_keys_object)
> > >
> > > Here's the error:
> > >
> > > DatabaseError: (raised as a result of Query-invoked autoflush;
> > consider
> > > using a session.no_autoflush block if this flush is occurring
> > > prematurely) (DatabaseError) 1300: Invalid utf8 character
> string:
> > > 'FE4587' u'INSERT INTO rotating_keys (gcm_key, nonce, tag,
> > > operational_team_id) VALUES (%(gcm_key)s, %(nonce)s, %(tag)s,
> > > %(operational_team_id)s)' {'gcm_key':
> > >
> >
> '\xfeE\x87\xe7\xc9\xe5\xec\xe0\x9c\xd6\x85\x11\xc7\xebd\xe3\x7f\xd9\xfel\xe6\x86"j\xbe=\xf4\xd7\x95\x99F\x8f',
>
>
> >
> > > 'nonce': 'o\xcb\x06\xe0\xe9\xech\xed\xed?T\xf4\xaf\x9a\xe1N',
> > > 'operational_team_id': 1, 'tag':
> > > ";p\xcce\xd2\xb8'\xf5\x89q\xc1\xa0\xfa\xff\x11\xf9"}
> >
> > when you deal with non-ascii values in Python, it's best to use
> an
> > encoding-neutral Python unicode object, in Py2K this is a string
> > like
> > u'some string'. The conversion to utf8 is done by the database
> > driver
> > when it is passed from your application to the driver.
> >
> > If that's not the problem here then you'd need to illustrate
> > many more
> > details, including version of python in use, database driver in
> > use, an
> > MCVE code example that we can run (
Re: [sqlalchemy] utf8 error upon insert
Any ideas?
On Thursday, June 9, 2016 at 10:21:27 AM UTC-4, Ven Karri wrote:
>
> Using, python 2.7 using 'mysql+pymysql' driver
>
> Code is very simple:
>
> gcm_key = '\xfeE\x87\xe7\xc9\xe5\xec\xe0\x9c\xd6\x85\x11\xc7\xebd\
> xe3\x7f\xd9\xfel\xe6\x86"j\xbe=\xf4\xd7\x95\x99F\x8f'
> model = Sample(gcm_key=gcm_key)
> session.add(model)
>
> Now what you said is to make it u'some string'. The string in question
> here is the gcm_key. So, I did this:
>
> gcm_key = gcm_key.decode('utf8')
>
> That throws an error:
>
> UnicodeDecodeError: 'utf8' codec can't decode byte 0xfe in position 0:
> invalid start byte
>
> On Thursday, June 9, 2016 at 10:04:49 AM UTC-4, Mike Bayer wrote:
>>
>>
>>
>> On 06/09/2016 09:52 AM, Ven Karri wrote:
>> > I am getting a UTF-8 error upon insert using sql alchemy ORM. The same
>> > query runs fine when I run using raw sql. Here's the ORM query:
>> >
>> > rotating_keys_object = rotating_keys_model(
>> > gcm_key=rot_gcm_key,
>> > nonce=rot_nonce,
>> > tag=rot_tag,
>> > operational_team_id=self.operational_team_id
>> > )
>> > session.add(rotating_keys_object)
>> >
>> > Here's the error:
>> >
>> > DatabaseError: (raised as a result of Query-invoked autoflush; consider
>> > using a session.no_autoflush block if this flush is occurring
>> > prematurely) (DatabaseError) 1300: Invalid utf8 character string:
>> > 'FE4587' u'INSERT INTO rotating_keys (gcm_key, nonce, tag,
>> > operational_team_id) VALUES (%(gcm_key)s, %(nonce)s, %(tag)s,
>> > %(operational_team_id)s)' {'gcm_key':
>> >
>> '\xfeE\x87\xe7\xc9\xe5\xec\xe0\x9c\xd6\x85\x11\xc7\xebd\xe3\x7f\xd9\xfel\xe6\x86"j\xbe=\xf4\xd7\x95\x99F\x8f',
>>
>>
>> > 'nonce': 'o\xcb\x06\xe0\xe9\xech\xed\xed?T\xf4\xaf\x9a\xe1N',
>> > 'operational_team_id': 1, 'tag':
>> > ";p\xcce\xd2\xb8'\xf5\x89q\xc1\xa0\xfa\xff\x11\xf9"}
>>
>> when you deal with non-ascii values in Python, it's best to use an
>> encoding-neutral Python unicode object, in Py2K this is a string like
>> u'some string'. The conversion to utf8 is done by the database driver
>> when it is passed from your application to the driver.
>>
>> If that's not the problem here then you'd need to illustrate many more
>> details, including version of python in use, database driver in use, an
>> MCVE code example that we can run (see http://stackoverflow.com/help/mcve).
>>
>>
>>
>>
>>
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups "sqlalchemy" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an email to sqlalchemy+...@googlegroups.com
>> > <mailto:sqlalchemy+unsubscr...@googlegroups.com>.
>> > To post to this group, send email to sqlal...@googlegroups.com
>> > <mailto:sqlal...@googlegroups.com>.
>> > Visit this group at https://groups.google.com/group/sqlalchemy.
>> > For more options, visit https://groups.google.com/d/optout.
>>
>
--
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
Re: [sqlalchemy] utf8 error upon insert
Using, python 2.7 using 'mysql+pymysql' driver
Code is very simple:
gcm_key = '\xfeE\x87\xe7\xc9\xe5\xec\xe0\x9c\xd6\x85\x11\xc7\xebd\
xe3\x7f\xd9\xfel\xe6\x86"j\xbe=\xf4\xd7\x95\x99F\x8f'
model = Sample(gcm_key=gcm_key)
session.add(model)
Now what you said is to make it u'some string'. The string in question here
is the gcm_key. So, I did this:
gcm_key = gcm_key.decode('utf8')
That throws an error:
UnicodeDecodeError: 'utf8' codec can't decode byte 0xfe in position 0:
invalid start byte
On Thursday, June 9, 2016 at 10:04:49 AM UTC-4, Mike Bayer wrote:
>
>
>
> On 06/09/2016 09:52 AM, Ven Karri wrote:
> > I am getting a UTF-8 error upon insert using sql alchemy ORM. The same
> > query runs fine when I run using raw sql. Here's the ORM query:
> >
> > rotating_keys_object = rotating_keys_model(
> > gcm_key=rot_gcm_key,
> > nonce=rot_nonce,
> > tag=rot_tag,
> > operational_team_id=self.operational_team_id
> > )
> > session.add(rotating_keys_object)
> >
> > Here's the error:
> >
> > DatabaseError: (raised as a result of Query-invoked autoflush; consider
> > using a session.no_autoflush block if this flush is occurring
> > prematurely) (DatabaseError) 1300: Invalid utf8 character string:
> > 'FE4587' u'INSERT INTO rotating_keys (gcm_key, nonce, tag,
> > operational_team_id) VALUES (%(gcm_key)s, %(nonce)s, %(tag)s,
> > %(operational_team_id)s)' {'gcm_key':
> >
> '\xfeE\x87\xe7\xc9\xe5\xec\xe0\x9c\xd6\x85\x11\xc7\xebd\xe3\x7f\xd9\xfel\xe6\x86"j\xbe=\xf4\xd7\x95\x99F\x8f',
>
>
> > 'nonce': 'o\xcb\x06\xe0\xe9\xech\xed\xed?T\xf4\xaf\x9a\xe1N',
> > 'operational_team_id': 1, 'tag':
> > ";p\xcce\xd2\xb8'\xf5\x89q\xc1\xa0\xfa\xff\x11\xf9"}
>
> when you deal with non-ascii values in Python, it's best to use an
> encoding-neutral Python unicode object, in Py2K this is a string like
> u'some string'. The conversion to utf8 is done by the database driver
> when it is passed from your application to the driver.
>
> If that's not the problem here then you'd need to illustrate many more
> details, including version of python in use, database driver in use, an
> MCVE code example that we can run (see http://stackoverflow.com/help/mcve).
>
>
>
>
>
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "sqlalchemy" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> > an email to sqlalchemy+...@googlegroups.com
> > <mailto:sqlalchemy+unsubscr...@googlegroups.com >.
> > To post to this group, send email to sqlal...@googlegroups.com
>
> > <mailto:sqlal...@googlegroups.com >.
> > Visit this group at https://groups.google.com/group/sqlalchemy.
> > For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
[sqlalchemy] Re: utf8 error upon insert
Btw, It's a MySQL backend
On Thursday, June 9, 2016 at 9:52:41 AM UTC-4, Ven Karri wrote:
>
> I am getting a UTF-8 error upon insert using sql alchemy ORM. The same
> query runs fine when I run using raw sql. Here's the ORM query:
>
> rotating_keys_object = rotating_keys_model(
> gcm_key=rot_gcm_key,
> nonce=rot_nonce,
> tag=rot_tag,
> operational_team_id=self.operational_team_id
> )
> session.add(rotating_keys_object)
>
> Here's the error:
>
> DatabaseError: (raised as a result of Query-invoked autoflush; consider
> using a session.no_autoflush block if this flush is occurring prematurely)
> (DatabaseError) 1300: Invalid utf8 character string: 'FE4587' u'INSERT INTO
> rotating_keys (gcm_key, nonce, tag, operational_team_id) VALUES
> (%(gcm_key)s, %(nonce)s, %(tag)s, %(operational_team_id)s)' {'gcm_key':
> '\xfeE\x87\xe7\xc9\xe5\xec\xe0\x9c\xd6\x85\x11\xc7\xebd\xe3\x7f\xd9\xfel\xe6\x86"j\xbe=\xf4\xd7\x95\x99F\x8f',
>
> 'nonce': 'o\xcb\x06\xe0\xe9\xech\xed\xed?T\xf4\xaf\x9a\xe1N',
> 'operational_team_id': 1, 'tag':
> ";p\xcce\xd2\xb8'\xf5\x89q\xc1\xa0\xfa\xff\x11\xf9"}
>
--
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
[sqlalchemy] utf8 error upon insert
I am getting a UTF-8 error upon insert using sql alchemy ORM. The same
query runs fine when I run using raw sql. Here's the ORM query:
rotating_keys_object = rotating_keys_model(
gcm_key=rot_gcm_key,
nonce=rot_nonce,
tag=rot_tag,
operational_team_id=self.operational_team_id
)
session.add(rotating_keys_object)
Here's the error:
DatabaseError: (raised as a result of Query-invoked autoflush; consider
using a session.no_autoflush block if this flush is occurring prematurely)
(DatabaseError) 1300: Invalid utf8 character string: 'FE4587' u'INSERT INTO
rotating_keys (gcm_key, nonce, tag, operational_team_id) VALUES
(%(gcm_key)s, %(nonce)s, %(tag)s, %(operational_team_id)s)' {'gcm_key':
'\xfeE\x87\xe7\xc9\xe5\xec\xe0\x9c\xd6\x85\x11\xc7\xebd\xe3\x7f\xd9\xfel\xe6\x86"j\xbe=\xf4\xd7\x95\x99F\x8f',
'nonce': 'o\xcb\x06\xe0\xe9\xech\xed\xed?T\xf4\xaf\x9a\xe1N',
'operational_team_id': 1, 'tag':
";p\xcce\xd2\xb8'\xf5\x89q\xc1\xa0\xfa\xff\x11\xf9"}
--
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
[sqlalchemy] Postgres OID column in SQLA 0.8
Hi Mike, Is there anyway, you can make the following changes to SQLA 0.8 ? https://bitbucket.org/zzzeek/sqlalchemy/commits/42bbb7163ada -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
[sqlalchemy] How to define metaclass for a class that extends from sqlalchemy declarative base ?
I use: Python 2.6 and sqlalchemy 0.6.1
This is what I am trying to do:
from sqlalchemy.types import (
Integer,
String,
Boolean
)
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class SampleMeta(type):
def __new__(cls, name, bases, attrs):
attrs.update({ 'id': Column('Id', Integer, primary_key=True),
'name': Column('Name', String),
'description': Column('Description', String),
'is_active': Column('IsActive', Boolean)
})
return super(SampleMeta, cls).__new__(cls, name, bases, attrs)
class Sample(Base):
__tablename__ = 'Sample'
__table_args__ = {'useexisting': True}
__metaclass__ = SampleMeta
def __init__(self, id, name, description, is_active):
self.id = id
self.name = name
self.description = description
self.is_active = is_active
def __repr__(self):
return (%d, '%s', '%s', %r) % (self.id, self.name,
self.description, self.isactive)
And the error I am getting is this:
TypeError: Error when calling the metaclass bases
metaclass conflict: the metaclass of a derived class must be a
(non-strict) subclass of the metaclasses of all its bases
Now, if I do the same thing above by using
class Sample(object)
instead of
class Sample(Base)
it works absolutely fine.
I need to update the attributes of the class dynamically. So, I will be
using dynamic attribute and column names. And I need the above piece code
to work in order to be able to get there.
**Please help**
--
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/groups/opt_out.
