Re: [sqlalchemy] utf8 error upon insert
upgrading pymysql did the trick. thank you.
On Thursday, June 9, 2016 at 12:02:29 PM UTC-4, Mike Bayer wrote:
>
>
> VARBINARY should not have a utf-8 encoding step at all. I can replace
> VARBINARY directly in my script and there's no problem; can you upgrade
> your pymysql? I seem to recall someone having this problem recently.
>
>
> Also please run the script below (with your database URL) and send the
> full output, as it will include things like the SQL MODE you're running
> and other things:
>
> from sqlalchemy import *
> from sqlalchemy.orm import *
> from sqlalchemy.ext.declarative import declarative_base
>
> Base = declarative_base()
>
>
> class A(Base):
> __tablename__ = 'a'
> id = Column(Integer, primary_key=True)
> data = Column(VARBINARY(255))
>
> e = create_engine("mysql+pymysql://scott:tiger@localhost/test", echo=True)
> Base.metadata.drop_all(e)
> Base.metadata.create_all(e)
>
> gcm_key =
> '\xfeE\x87\xe7\xc9\xe5\xec\xe0\x9c\xd6\x85\x11\xc7\xebd\xe3\x7f\xd9\xfel\xe6\x86"j\xbe=\xf4\xd7\x95\x99F\x8f'
>
>
>
> s = Session(e)
> s.add(A(data=gcm_key))
> s.commit()
>
>
>
> On 06/09/2016 11:16 AM, Ven Karri wrote:
> > The only difference is that I am using sqlalchemy.types.VARBINARY(256)
> > instead of String(255)
> >
> > On Thursday, June 9, 2016 at 10:47:32 AM UTC-4, Mike Bayer wrote:
> >
> > what charset is in your my.cnf and/or how are you connecting. Mine
> > only
> > produces a warning. Here is an MCVE (definition: I can actually run
> > it):
> >
> > from sqlalchemy import *
> > from sqlalchemy.orm import *
> > from sqlalchemy.ext.declarative import declarative_base
> >
> > Base = declarative_base()
> >
> >
> > class A(Base):
> > __tablename__ = 'a'
> > id = Column(Integer, primary_key=True)
> > data = Column(String(255))
> >
> > e = create_engine("mysql+pymysql://scott:tiger@localhost/test",
> > echo=True)
> > Base.metadata.create_all(e)
> >
> > gcm_key =
> >
> '\xfeE\x87\xe7\xc9\xe5\xec\xe0\x9c\xd6\x85\x11\xc7\xebd\xe3\x7f\xd9\xfel\xe6\x86"j\xbe=\xf4\xd7\x95\x99F\x8f'
>
>
> >
> >
> > s = Session(e)
> > s.add(A(data=gcm_key))
> > s.commit()
> >
> >
> >
> >
> > output:
> >
> > python test.py
> >
> /home/classic/.venv/lib/python2.7/site-packages/pymysql/cursors.py:146:
> > Warning: Table 'test' already exists
> >result = self._query(query)
> >
> /home/classic/.venv/lib/python2.7/site-packages/pymysql/cursors.py:146:
> > Warning: Incorrect string value: '\xFEE\x87\xE7\xC9\xE5...' for
> column
> > 'data' at row 1
> >result = self._query(query)
> >
> >
> > please run this script and modify it to show your error thanks!
> >
> >
> >
> >
> >
> > On 06/09/2016 10:42 AM, Ven Karri wrote:
> > > Any ideas?
> > >
> > > On Thursday, June 9, 2016 at 10:21:27 AM UTC-4, Ven Karri wrote:
> > >
> > > Using, python 2.7 using 'mysql+pymysql' driver
> > >
> > > Code is very simple:
> > >
> > > gcm_key =
> > >
> >
> '\xfeE\x87\xe7\xc9\xe5\xec\xe0\x9c\xd6\x85\x11\xc7\xebd\xe3\x7f\xd9\xfel\xe6\x86"j\xbe=\xf4\xd7\x95\x99F\x8f'
>
>
> >
> > > model = Sample(gcm_key=gcm_key)
> > > session.add(model)
> > >
> > > Now what you said is to make it u'some string'. The string in
> > > question here is the gcm_key. So, I did this:
> > >
> > > gcm_key = gcm_key.decode('utf8')
> > >
> > > That throws an error:
> > >
> > > UnicodeDecodeError: 'utf8' codec can't decode byte 0xfe in
> > position
> > > 0: invalid start byte
> > >
> > > On Thursday, June 9, 2016 at 10:04:49 AM UTC-4, Mike Bayer
> wrote:
> > >
> > >
> > >
> > > On 06/09/2016 09:52 AM, Ven Karri wrote:
> > > > I am getting a UTF-8 error upon insert using sql alchemy
> > ORM.
> > > The same
> > > > query runs fine when I run using raw sql. Here's the ORM
> > query:
> > > >
> > > > rotating_keys_object = rotating_keys_model(
> > > > gcm_key=rot_gcm_key,
> > > > nonce=rot_nonce,
> > > > tag=rot_tag,
> > > > operational_team_id=self.operational_team_id
> > > > )
> > > > session.add(rotating_keys_object)
> > > >
> > > > Here's the error:
> > > >
> > > > DatabaseError: (raised as a result of Query-invoked
> > autoflush;
> > > consider
> > > > using a session.no_autoflush block if this flush is
> > occurring
> > > > prematurely) (DatabaseError) 1300: Invalid utf8
> > character string:
> > > > 'FE4587'
Re: [sqlalchemy] utf8 error upon insert
VARBINARY should not have a utf-8 encoding step at all. I can replace
VARBINARY directly in my script and there's no problem; can you upgrade
your pymysql? I seem to recall someone having this problem recently.
Also please run the script below (with your database URL) and send the
full output, as it will include things like the SQL MODE you're running
and other things:
from sqlalchemy import *
from sqlalchemy.orm import *
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class A(Base):
__tablename__ = 'a'
id = Column(Integer, primary_key=True)
data = Column(VARBINARY(255))
e = create_engine("mysql+pymysql://scott:tiger@localhost/test", echo=True)
Base.metadata.drop_all(e)
Base.metadata.create_all(e)
gcm_key =
'\xfeE\x87\xe7\xc9\xe5\xec\xe0\x9c\xd6\x85\x11\xc7\xebd\xe3\x7f\xd9\xfel\xe6\x86"j\xbe=\xf4\xd7\x95\x99F\x8f'
s = Session(e)
s.add(A(data=gcm_key))
s.commit()
On 06/09/2016 11:16 AM, Ven Karri wrote:
The only difference is that I am using sqlalchemy.types.VARBINARY(256)
instead of String(255)
On Thursday, June 9, 2016 at 10:47:32 AM UTC-4, Mike Bayer wrote:
what charset is in your my.cnf and/or how are you connecting. Mine
only
produces a warning. Here is an MCVE (definition: I can actually run
it):
from sqlalchemy import *
from sqlalchemy.orm import *
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class A(Base):
__tablename__ = 'a'
id = Column(Integer, primary_key=True)
data = Column(String(255))
e = create_engine("mysql+pymysql://scott:tiger@localhost/test",
echo=True)
Base.metadata.create_all(e)
gcm_key =
'\xfeE\x87\xe7\xc9\xe5\xec\xe0\x9c\xd6\x85\x11\xc7\xebd\xe3\x7f\xd9\xfel\xe6\x86"j\xbe=\xf4\xd7\x95\x99F\x8f'
s = Session(e)
s.add(A(data=gcm_key))
s.commit()
output:
python test.py
/home/classic/.venv/lib/python2.7/site-packages/pymysql/cursors.py:146:
Warning: Table 'test' already exists
result = self._query(query)
/home/classic/.venv/lib/python2.7/site-packages/pymysql/cursors.py:146:
Warning: Incorrect string value: '\xFEE\x87\xE7\xC9\xE5...' for column
'data' at row 1
result = self._query(query)
please run this script and modify it to show your error thanks!
On 06/09/2016 10:42 AM, Ven Karri wrote:
> Any ideas?
>
> On Thursday, June 9, 2016 at 10:21:27 AM UTC-4, Ven Karri wrote:
>
> Using, python 2.7 using 'mysql+pymysql' driver
>
> Code is very simple:
>
> gcm_key =
>
'\xfeE\x87\xe7\xc9\xe5\xec\xe0\x9c\xd6\x85\x11\xc7\xebd\xe3\x7f\xd9\xfel\xe6\x86"j\xbe=\xf4\xd7\x95\x99F\x8f'
> model = Sample(gcm_key=gcm_key)
> session.add(model)
>
> Now what you said is to make it u'some string'. The string in
> question here is the gcm_key. So, I did this:
>
> gcm_key = gcm_key.decode('utf8')
>
> That throws an error:
>
> UnicodeDecodeError: 'utf8' codec can't decode byte 0xfe in
position
> 0: invalid start byte
>
> On Thursday, June 9, 2016 at 10:04:49 AM UTC-4, Mike Bayer wrote:
>
>
>
> On 06/09/2016 09:52 AM, Ven Karri wrote:
> > I am getting a UTF-8 error upon insert using sql alchemy
ORM.
> The same
> > query runs fine when I run using raw sql. Here's the ORM
query:
> >
> > rotating_keys_object = rotating_keys_model(
> > gcm_key=rot_gcm_key,
> > nonce=rot_nonce,
> > tag=rot_tag,
> > operational_team_id=self.operational_team_id
> > )
> > session.add(rotating_keys_object)
> >
> > Here's the error:
> >
> > DatabaseError: (raised as a result of Query-invoked
autoflush;
> consider
> > using a session.no_autoflush block if this flush is
occurring
> > prematurely) (DatabaseError) 1300: Invalid utf8
character string:
> > 'FE4587' u'INSERT INTO rotating_keys (gcm_key, nonce, tag,
> > operational_team_id) VALUES (%(gcm_key)s, %(nonce)s,
%(tag)s,
> > %(operational_team_id)s)' {'gcm_key':
> >
>
'\xfeE\x87\xe7\xc9\xe5\xec\xe0\x9c\xd6\x85\x11\xc7\xebd\xe3\x7f\xd9\xfel\xe6\x86"j\xbe=\xf4\xd7\x95\x99F\x8f',
>
> > 'nonce':
'o\xcb\x06\xe0\xe9\xech\xed\xed?T\xf4\xaf\x9a\xe1N',
> > 'operational_team_id': 1, 'tag':
> > ";p\xcce\xd2\xb8'\xf5\x89q\xc1\xa0\xfa\xff\x11\xf9"}
>
> when you deal with non-ascii values in Python, it's best
to use an
> encoding-neutral Python unicode object, in Py2K this is a
string
> like
> u'some
Re: [sqlalchemy] utf8 error upon insert
The only difference is that I am using sqlalchemy.types.VARBINARY(256)
instead of String(255)
On Thursday, June 9, 2016 at 10:47:32 AM UTC-4, Mike Bayer wrote:
>
> what charset is in your my.cnf and/or how are you connecting. Mine only
> produces a warning. Here is an MCVE (definition: I can actually run it):
>
> from sqlalchemy import *
> from sqlalchemy.orm import *
> from sqlalchemy.ext.declarative import declarative_base
>
> Base = declarative_base()
>
>
> class A(Base):
> __tablename__ = 'a'
> id = Column(Integer, primary_key=True)
> data = Column(String(255))
>
> e = create_engine("mysql+pymysql://scott:tiger@localhost/test", echo=True)
> Base.metadata.create_all(e)
>
> gcm_key =
> '\xfeE\x87\xe7\xc9\xe5\xec\xe0\x9c\xd6\x85\x11\xc7\xebd\xe3\x7f\xd9\xfel\xe6\x86"j\xbe=\xf4\xd7\x95\x99F\x8f'
>
>
>
> s = Session(e)
> s.add(A(data=gcm_key))
> s.commit()
>
>
>
>
> output:
>
> python test.py
> /home/classic/.venv/lib/python2.7/site-packages/pymysql/cursors.py:146:
> Warning: Table 'test' already exists
>result = self._query(query)
> /home/classic/.venv/lib/python2.7/site-packages/pymysql/cursors.py:146:
> Warning: Incorrect string value: '\xFEE\x87\xE7\xC9\xE5...' for column
> 'data' at row 1
>result = self._query(query)
>
>
> please run this script and modify it to show your error thanks!
>
>
>
>
>
> On 06/09/2016 10:42 AM, Ven Karri wrote:
> > Any ideas?
> >
> > On Thursday, June 9, 2016 at 10:21:27 AM UTC-4, Ven Karri wrote:
> >
> > Using, python 2.7 using 'mysql+pymysql' driver
> >
> > Code is very simple:
> >
> > gcm_key =
> >
> '\xfeE\x87\xe7\xc9\xe5\xec\xe0\x9c\xd6\x85\x11\xc7\xebd\xe3\x7f\xd9\xfel\xe6\x86"j\xbe=\xf4\xd7\x95\x99F\x8f'
>
>
> > model = Sample(gcm_key=gcm_key)
> > session.add(model)
> >
> > Now what you said is to make it u'some string'. The string in
> > question here is the gcm_key. So, I did this:
> >
> > gcm_key = gcm_key.decode('utf8')
> >
> > That throws an error:
> >
> > UnicodeDecodeError: 'utf8' codec can't decode byte 0xfe in position
> > 0: invalid start byte
> >
> > On Thursday, June 9, 2016 at 10:04:49 AM UTC-4, Mike Bayer wrote:
> >
> >
> >
> > On 06/09/2016 09:52 AM, Ven Karri wrote:
> > > I am getting a UTF-8 error upon insert using sql alchemy ORM.
> > The same
> > > query runs fine when I run using raw sql. Here's the ORM
> query:
> > >
> > > rotating_keys_object = rotating_keys_model(
> > > gcm_key=rot_gcm_key,
> > > nonce=rot_nonce,
> > > tag=rot_tag,
> > > operational_team_id=self.operational_team_id
> > > )
> > > session.add(rotating_keys_object)
> > >
> > > Here's the error:
> > >
> > > DatabaseError: (raised as a result of Query-invoked autoflush;
> > consider
> > > using a session.no_autoflush block if this flush is occurring
> > > prematurely) (DatabaseError) 1300: Invalid utf8 character
> string:
> > > 'FE4587' u'INSERT INTO rotating_keys (gcm_key, nonce, tag,
> > > operational_team_id) VALUES (%(gcm_key)s, %(nonce)s, %(tag)s,
> > > %(operational_team_id)s)' {'gcm_key':
> > >
> >
> '\xfeE\x87\xe7\xc9\xe5\xec\xe0\x9c\xd6\x85\x11\xc7\xebd\xe3\x7f\xd9\xfel\xe6\x86"j\xbe=\xf4\xd7\x95\x99F\x8f',
>
>
> >
> > > 'nonce': 'o\xcb\x06\xe0\xe9\xech\xed\xed?T\xf4\xaf\x9a\xe1N',
> > > 'operational_team_id': 1, 'tag':
> > > ";p\xcce\xd2\xb8'\xf5\x89q\xc1\xa0\xfa\xff\x11\xf9"}
> >
> > when you deal with non-ascii values in Python, it's best to use
> an
> > encoding-neutral Python unicode object, in Py2K this is a string
> > like
> > u'some string'. The conversion to utf8 is done by the database
> > driver
> > when it is passed from your application to the driver.
> >
> > If that's not the problem here then you'd need to illustrate
> > many more
> > details, including version of python in use, database driver in
> > use, an
> > MCVE code example that we can run (see
> > http://stackoverflow.com/help/mcve
> > ).
> >
> >
> >
> >
> > >
> > > --
> > > You received this message because you are subscribed to the
> > Google
> > > Groups "sqlalchemy" group.
> > > To unsubscribe from this group and stop receiving emails from
> > it, send
> > > an email to sqlalchemy+...@googlegroups.com
> > > .
>
> > > To post to this group, send email to sqlal...@googlegroups.com
> > > .
> > > Visit this group at
Re: [sqlalchemy] utf8 error upon insert
what charset is in your my.cnf and/or how are you connecting. Mine only
produces a warning. Here is an MCVE (definition: I can actually run it):
from sqlalchemy import *
from sqlalchemy.orm import *
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class A(Base):
__tablename__ = 'a'
id = Column(Integer, primary_key=True)
data = Column(String(255))
e = create_engine("mysql+pymysql://scott:tiger@localhost/test", echo=True)
Base.metadata.create_all(e)
gcm_key =
'\xfeE\x87\xe7\xc9\xe5\xec\xe0\x9c\xd6\x85\x11\xc7\xebd\xe3\x7f\xd9\xfel\xe6\x86"j\xbe=\xf4\xd7\x95\x99F\x8f'
s = Session(e)
s.add(A(data=gcm_key))
s.commit()
output:
python test.py
/home/classic/.venv/lib/python2.7/site-packages/pymysql/cursors.py:146:
Warning: Table 'test' already exists
result = self._query(query)
/home/classic/.venv/lib/python2.7/site-packages/pymysql/cursors.py:146:
Warning: Incorrect string value: '\xFEE\x87\xE7\xC9\xE5...' for column
'data' at row 1
result = self._query(query)
please run this script and modify it to show your error thanks!
On 06/09/2016 10:42 AM, Ven Karri wrote:
Any ideas?
On Thursday, June 9, 2016 at 10:21:27 AM UTC-4, Ven Karri wrote:
Using, python 2.7 using 'mysql+pymysql' driver
Code is very simple:
gcm_key =
'\xfeE\x87\xe7\xc9\xe5\xec\xe0\x9c\xd6\x85\x11\xc7\xebd\xe3\x7f\xd9\xfel\xe6\x86"j\xbe=\xf4\xd7\x95\x99F\x8f'
model = Sample(gcm_key=gcm_key)
session.add(model)
Now what you said is to make it u'some string'. The string in
question here is the gcm_key. So, I did this:
gcm_key = gcm_key.decode('utf8')
That throws an error:
UnicodeDecodeError: 'utf8' codec can't decode byte 0xfe in position
0: invalid start byte
On Thursday, June 9, 2016 at 10:04:49 AM UTC-4, Mike Bayer wrote:
On 06/09/2016 09:52 AM, Ven Karri wrote:
> I am getting a UTF-8 error upon insert using sql alchemy ORM.
The same
> query runs fine when I run using raw sql. Here's the ORM query:
>
> rotating_keys_object = rotating_keys_model(
> gcm_key=rot_gcm_key,
> nonce=rot_nonce,
> tag=rot_tag,
> operational_team_id=self.operational_team_id
> )
> session.add(rotating_keys_object)
>
> Here's the error:
>
> DatabaseError: (raised as a result of Query-invoked autoflush;
consider
> using a session.no_autoflush block if this flush is occurring
> prematurely) (DatabaseError) 1300: Invalid utf8 character string:
> 'FE4587' u'INSERT INTO rotating_keys (gcm_key, nonce, tag,
> operational_team_id) VALUES (%(gcm_key)s, %(nonce)s, %(tag)s,
> %(operational_team_id)s)' {'gcm_key':
>
'\xfeE\x87\xe7\xc9\xe5\xec\xe0\x9c\xd6\x85\x11\xc7\xebd\xe3\x7f\xd9\xfel\xe6\x86"j\xbe=\xf4\xd7\x95\x99F\x8f',
> 'nonce': 'o\xcb\x06\xe0\xe9\xech\xed\xed?T\xf4\xaf\x9a\xe1N',
> 'operational_team_id': 1, 'tag':
> ";p\xcce\xd2\xb8'\xf5\x89q\xc1\xa0\xfa\xff\x11\xf9"}
when you deal with non-ascii values in Python, it's best to use an
encoding-neutral Python unicode object, in Py2K this is a string
like
u'some string'. The conversion to utf8 is done by the database
driver
when it is passed from your application to the driver.
If that's not the problem here then you'd need to illustrate
many more
details, including version of python in use, database driver in
use, an
MCVE code example that we can run (see
http://stackoverflow.com/help/mcve
).
>
> --
> You received this message because you are subscribed to the
Google
> Groups "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from
it, send
> an email to sqlalchemy+...@googlegroups.com
> .
> To post to this group, send email to sqlal...@googlegroups.com
> .
> Visit this group at https://groups.google.com/group/sqlalchemy
.
> For more options, visit https://groups.google.com/d/optout
.
--
You received this message because you are subscribed to the Google
Groups "sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to sqlalchemy+unsubscr...@googlegroups.com
.
To post to this group, send email to sqlalchemy@googlegroups.com
.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
Re: [sqlalchemy] utf8 error upon insert
On 06/09/2016 10:21 AM, Ven Karri wrote:
Using, python 2.7 using 'mysql+pymysql' driver
Code is very simple:
gcm_key =
'\xfeE\x87\xe7\xc9\xe5\xec\xe0\x9c\xd6\x85\x11\xc7\xebd\xe3\x7f\xd9\xfel\xe6\x86"j\xbe=\xf4\xd7\x95\x99F\x8f'
model = Sample(gcm_key=gcm_key)
session.add(model)
Now what you said is to make it u'some string'. The string in question
here is the gcm_key. So, I did this:
gcm_key = gcm_key.decode('utf8')
That throws an error:
UnicodeDecodeError: 'utf8' codec can't decode byte 0xfe in position 0:
invalid start byte
so the data you're starting out with is not valid utf8. That would be
your problem.
On Thursday, June 9, 2016 at 10:04:49 AM UTC-4, Mike Bayer wrote:
On 06/09/2016 09:52 AM, Ven Karri wrote:
> I am getting a UTF-8 error upon insert using sql alchemy ORM. The
same
> query runs fine when I run using raw sql. Here's the ORM query:
>
> rotating_keys_object = rotating_keys_model(
> gcm_key=rot_gcm_key,
> nonce=rot_nonce,
> tag=rot_tag,
> operational_team_id=self.operational_team_id
> )
> session.add(rotating_keys_object)
>
> Here's the error:
>
> DatabaseError: (raised as a result of Query-invoked autoflush;
consider
> using a session.no_autoflush block if this flush is occurring
> prematurely) (DatabaseError) 1300: Invalid utf8 character string:
> 'FE4587' u'INSERT INTO rotating_keys (gcm_key, nonce, tag,
> operational_team_id) VALUES (%(gcm_key)s, %(nonce)s, %(tag)s,
> %(operational_team_id)s)' {'gcm_key':
>
'\xfeE\x87\xe7\xc9\xe5\xec\xe0\x9c\xd6\x85\x11\xc7\xebd\xe3\x7f\xd9\xfel\xe6\x86"j\xbe=\xf4\xd7\x95\x99F\x8f',
> 'nonce': 'o\xcb\x06\xe0\xe9\xech\xed\xed?T\xf4\xaf\x9a\xe1N',
> 'operational_team_id': 1, 'tag':
> ";p\xcce\xd2\xb8'\xf5\x89q\xc1\xa0\xfa\xff\x11\xf9"}
when you deal with non-ascii values in Python, it's best to use an
encoding-neutral Python unicode object, in Py2K this is a string like
u'some string'. The conversion to utf8 is done by the database driver
when it is passed from your application to the driver.
If that's not the problem here then you'd need to illustrate many more
details, including version of python in use, database driver in use, an
MCVE code example that we can run (see
http://stackoverflow.com/help/mcve
).
>
> --
> You received this message because you are subscribed to the Google
> Groups "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it,
send
> an email to sqlalchemy+...@googlegroups.com
> .
> To post to this group, send email to sqlal...@googlegroups.com
> .
> Visit this group at https://groups.google.com/group/sqlalchemy
.
> For more options, visit https://groups.google.com/d/optout
.
--
You received this message because you are subscribed to the Google
Groups "sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to sqlalchemy+unsubscr...@googlegroups.com
.
To post to this group, send email to sqlalchemy@googlegroups.com
.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
Re: [sqlalchemy] utf8 error upon insert
Any ideas?
On Thursday, June 9, 2016 at 10:21:27 AM UTC-4, Ven Karri wrote:
>
> Using, python 2.7 using 'mysql+pymysql' driver
>
> Code is very simple:
>
> gcm_key = '\xfeE\x87\xe7\xc9\xe5\xec\xe0\x9c\xd6\x85\x11\xc7\xebd\
> xe3\x7f\xd9\xfel\xe6\x86"j\xbe=\xf4\xd7\x95\x99F\x8f'
> model = Sample(gcm_key=gcm_key)
> session.add(model)
>
> Now what you said is to make it u'some string'. The string in question
> here is the gcm_key. So, I did this:
>
> gcm_key = gcm_key.decode('utf8')
>
> That throws an error:
>
> UnicodeDecodeError: 'utf8' codec can't decode byte 0xfe in position 0:
> invalid start byte
>
> On Thursday, June 9, 2016 at 10:04:49 AM UTC-4, Mike Bayer wrote:
>>
>>
>>
>> On 06/09/2016 09:52 AM, Ven Karri wrote:
>> > I am getting a UTF-8 error upon insert using sql alchemy ORM. The same
>> > query runs fine when I run using raw sql. Here's the ORM query:
>> >
>> > rotating_keys_object = rotating_keys_model(
>> > gcm_key=rot_gcm_key,
>> > nonce=rot_nonce,
>> > tag=rot_tag,
>> > operational_team_id=self.operational_team_id
>> > )
>> > session.add(rotating_keys_object)
>> >
>> > Here's the error:
>> >
>> > DatabaseError: (raised as a result of Query-invoked autoflush; consider
>> > using a session.no_autoflush block if this flush is occurring
>> > prematurely) (DatabaseError) 1300: Invalid utf8 character string:
>> > 'FE4587' u'INSERT INTO rotating_keys (gcm_key, nonce, tag,
>> > operational_team_id) VALUES (%(gcm_key)s, %(nonce)s, %(tag)s,
>> > %(operational_team_id)s)' {'gcm_key':
>> >
>> '\xfeE\x87\xe7\xc9\xe5\xec\xe0\x9c\xd6\x85\x11\xc7\xebd\xe3\x7f\xd9\xfel\xe6\x86"j\xbe=\xf4\xd7\x95\x99F\x8f',
>>
>>
>> > 'nonce': 'o\xcb\x06\xe0\xe9\xech\xed\xed?T\xf4\xaf\x9a\xe1N',
>> > 'operational_team_id': 1, 'tag':
>> > ";p\xcce\xd2\xb8'\xf5\x89q\xc1\xa0\xfa\xff\x11\xf9"}
>>
>> when you deal with non-ascii values in Python, it's best to use an
>> encoding-neutral Python unicode object, in Py2K this is a string like
>> u'some string'. The conversion to utf8 is done by the database driver
>> when it is passed from your application to the driver.
>>
>> If that's not the problem here then you'd need to illustrate many more
>> details, including version of python in use, database driver in use, an
>> MCVE code example that we can run (see http://stackoverflow.com/help/mcve).
>>
>>
>>
>>
>>
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups "sqlalchemy" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an email to sqlalchemy+...@googlegroups.com
>> > .
>> > To post to this group, send email to sqlal...@googlegroups.com
>> > .
>> > Visit this group at https://groups.google.com/group/sqlalchemy.
>> > For more options, visit https://groups.google.com/d/optout.
>>
>
--
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
Re: [sqlalchemy] utf8 error upon insert
Using, python 2.7 using 'mysql+pymysql' driver
Code is very simple:
gcm_key = '\xfeE\x87\xe7\xc9\xe5\xec\xe0\x9c\xd6\x85\x11\xc7\xebd\
xe3\x7f\xd9\xfel\xe6\x86"j\xbe=\xf4\xd7\x95\x99F\x8f'
model = Sample(gcm_key=gcm_key)
session.add(model)
Now what you said is to make it u'some string'. The string in question here
is the gcm_key. So, I did this:
gcm_key = gcm_key.decode('utf8')
That throws an error:
UnicodeDecodeError: 'utf8' codec can't decode byte 0xfe in position 0:
invalid start byte
On Thursday, June 9, 2016 at 10:04:49 AM UTC-4, Mike Bayer wrote:
>
>
>
> On 06/09/2016 09:52 AM, Ven Karri wrote:
> > I am getting a UTF-8 error upon insert using sql alchemy ORM. The same
> > query runs fine when I run using raw sql. Here's the ORM query:
> >
> > rotating_keys_object = rotating_keys_model(
> > gcm_key=rot_gcm_key,
> > nonce=rot_nonce,
> > tag=rot_tag,
> > operational_team_id=self.operational_team_id
> > )
> > session.add(rotating_keys_object)
> >
> > Here's the error:
> >
> > DatabaseError: (raised as a result of Query-invoked autoflush; consider
> > using a session.no_autoflush block if this flush is occurring
> > prematurely) (DatabaseError) 1300: Invalid utf8 character string:
> > 'FE4587' u'INSERT INTO rotating_keys (gcm_key, nonce, tag,
> > operational_team_id) VALUES (%(gcm_key)s, %(nonce)s, %(tag)s,
> > %(operational_team_id)s)' {'gcm_key':
> >
> '\xfeE\x87\xe7\xc9\xe5\xec\xe0\x9c\xd6\x85\x11\xc7\xebd\xe3\x7f\xd9\xfel\xe6\x86"j\xbe=\xf4\xd7\x95\x99F\x8f',
>
>
> > 'nonce': 'o\xcb\x06\xe0\xe9\xech\xed\xed?T\xf4\xaf\x9a\xe1N',
> > 'operational_team_id': 1, 'tag':
> > ";p\xcce\xd2\xb8'\xf5\x89q\xc1\xa0\xfa\xff\x11\xf9"}
>
> when you deal with non-ascii values in Python, it's best to use an
> encoding-neutral Python unicode object, in Py2K this is a string like
> u'some string'. The conversion to utf8 is done by the database driver
> when it is passed from your application to the driver.
>
> If that's not the problem here then you'd need to illustrate many more
> details, including version of python in use, database driver in use, an
> MCVE code example that we can run (see http://stackoverflow.com/help/mcve).
>
>
>
>
>
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "sqlalchemy" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> > an email to sqlalchemy+...@googlegroups.com
> > .
> > To post to this group, send email to sqlal...@googlegroups.com
>
> > .
> > Visit this group at https://groups.google.com/group/sqlalchemy.
> > For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
Re: [sqlalchemy] utf8 error upon insert
On 06/09/2016 09:52 AM, Ven Karri wrote:
I am getting a UTF-8 error upon insert using sql alchemy ORM. The same
query runs fine when I run using raw sql. Here's the ORM query:
rotating_keys_object = rotating_keys_model(
gcm_key=rot_gcm_key,
nonce=rot_nonce,
tag=rot_tag,
operational_team_id=self.operational_team_id
)
session.add(rotating_keys_object)
Here's the error:
DatabaseError: (raised as a result of Query-invoked autoflush; consider
using a session.no_autoflush block if this flush is occurring
prematurely) (DatabaseError) 1300: Invalid utf8 character string:
'FE4587' u'INSERT INTO rotating_keys (gcm_key, nonce, tag,
operational_team_id) VALUES (%(gcm_key)s, %(nonce)s, %(tag)s,
%(operational_team_id)s)' {'gcm_key':
'\xfeE\x87\xe7\xc9\xe5\xec\xe0\x9c\xd6\x85\x11\xc7\xebd\xe3\x7f\xd9\xfel\xe6\x86"j\xbe=\xf4\xd7\x95\x99F\x8f',
'nonce': 'o\xcb\x06\xe0\xe9\xech\xed\xed?T\xf4\xaf\x9a\xe1N',
'operational_team_id': 1, 'tag':
";p\xcce\xd2\xb8'\xf5\x89q\xc1\xa0\xfa\xff\x11\xf9"}
when you deal with non-ascii values in Python, it's best to use an
encoding-neutral Python unicode object, in Py2K this is a string like
u'some string'. The conversion to utf8 is done by the database driver
when it is passed from your application to the driver.
If that's not the problem here then you'd need to illustrate many more
details, including version of python in use, database driver in use, an
MCVE code example that we can run (see http://stackoverflow.com/help/mcve).
--
You received this message because you are subscribed to the Google
Groups "sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to sqlalchemy+unsubscr...@googlegroups.com
.
To post to this group, send email to sqlalchemy@googlegroups.com
.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
