VARBINARY should not have a utf-8 encoding step at all. I can replace
VARBINARY directly in my script and there's no problem; can you upgrade
your pymysql? I seem to recall someone having this problem recently.
Also please run the script below (with your database URL) and send the
full output, as it will include things like the SQL MODE you're running
and other things:
from sqlalchemy import *
from sqlalchemy.orm import *
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class A(Base):
__tablename__ = 'a'
id = Column(Integer, primary_key=True)
data = Column(VARBINARY(255))
e = create_engine("mysql+pymysql://scott:tiger@localhost/test", echo=True)
Base.metadata.drop_all(e)
Base.metadata.create_all(e)
gcm_key =
'\xfeE\x87\xe7\xc9\xe5\xec\xe0\x9c\xd6\x85\x11\xc7\xebd\xe3\x7f\xd9\xfel\xe6\x86"j\xbe=\xf4\xd7\x95\x99F\x8f'
s = Session(e)
s.add(A(data=gcm_key))
s.commit()
On 06/09/2016 11:16 AM, Ven Karri wrote:
The only difference is that I am using sqlalchemy.types.VARBINARY(256)
instead of String(255)
On Thursday, June 9, 2016 at 10:47:32 AM UTC-4, Mike Bayer wrote:
what charset is in your my.cnf and/or how are you connecting. Mine
only
produces a warning. Here is an MCVE (definition: I can actually run
it):
from sqlalchemy import *
from sqlalchemy.orm import *
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class A(Base):
__tablename__ = 'a'
id = Column(Integer, primary_key=True)
data = Column(String(255))
e = create_engine("mysql+pymysql://scott:tiger@localhost/test",
echo=True)
Base.metadata.create_all(e)
gcm_key =
'\xfeE\x87\xe7\xc9\xe5\xec\xe0\x9c\xd6\x85\x11\xc7\xebd\xe3\x7f\xd9\xfel\xe6\x86"j\xbe=\xf4\xd7\x95\x99F\x8f'
s = Session(e)
s.add(A(data=gcm_key))
s.commit()
output:
python test.py
/home/classic/.venv/lib/python2.7/site-packages/pymysql/cursors.py:146:
Warning: Table 'test' already exists
result = self._query(query)
/home/classic/.venv/lib/python2.7/site-packages/pymysql/cursors.py:146:
Warning: Incorrect string value: '\xFEE\x87\xE7\xC9\xE5...' for column
'data' at row 1
result = self._query(query)
please run this script and modify it to show your error thanks!
On 06/09/2016 10:42 AM, Ven Karri wrote:
> Any ideas?
>
> On Thursday, June 9, 2016 at 10:21:27 AM UTC-4, Ven Karri wrote:
>
> Using, python 2.7 using 'mysql+pymysql' driver
>
> Code is very simple:
>
> gcm_key =
>
'\xfeE\x87\xe7\xc9\xe5\xec\xe0\x9c\xd6\x85\x11\xc7\xebd\xe3\x7f\xd9\xfel\xe6\x86"j\xbe=\xf4\xd7\x95\x99F\x8f'
> model = Sample(gcm_key=gcm_key)
> session.add(model)
>
> Now what you said is to make it u'some string'. The string in
> question here is the gcm_key. So, I did this:
>
> gcm_key = gcm_key.decode('utf8')
>
> That throws an error:
>
> UnicodeDecodeError: 'utf8' codec can't decode byte 0xfe in
position
> 0: invalid start byte
>
> On Thursday, June 9, 2016 at 10:04:49 AM UTC-4, Mike Bayer wrote:
>
>
>
> On 06/09/2016 09:52 AM, Ven Karri wrote:
> > I am getting a UTF-8 error upon insert using sql alchemy
ORM.
> The same
> > query runs fine when I run using raw sql. Here's the ORM
query:
> >
> > rotating_keys_object = rotating_keys_model(
> > gcm_key=rot_gcm_key,
> > nonce=rot_nonce,
> > tag=rot_tag,
> > operational_team_id=self.operational_team_id
> > )
> > session.add(rotating_keys_object)
> >
> > Here's the error:
> >
> > DatabaseError: (raised as a result of Query-invoked
autoflush;
> consider
> > using a session.no_autoflush block if this flush is
occurring
> > prematurely) (DatabaseError) 1300: Invalid utf8
character string:
> > 'FE4587' u'INSERT INTO rotating_keys (gcm_key, nonce, tag,
> > operational_team_id) VALUES (%(gcm_key)s, %(nonce)s,
%(tag)s,
> > %(operational_team_id)s)' {'gcm_key':
> >
>
'\xfeE\x87\xe7\xc9\xe5\xec\xe0\x9c\xd6\x85\x11\xc7\xebd\xe3\x7f\xd9\xfel\xe6\x86"j\xbe=\xf4\xd7\x95\x99F\x8f',
>
> > 'nonce':
'o\xcb\x06\xe0\xe9\xech\xed\xed?T\xf4\xaf\x9a\xe1N',
> > 'operational_team_id': 1, 'tag':
> > ";p\xcce\xd2\xb8'\xf5\x89q\xc1\xa0\xfa\xff\x11\xf9"}
>
> when you deal with non-ascii values in Python, it's best
to use an
> encoding-neutral Python unicode object, in Py2K this is a
string
> like
> u'some string'. The conversion to utf8 is done by the
database
> driver
> when it is passed from your application to the driver.
>
> If that's not the problem here then you'd need to illustrate
> many more
> details, including version of python in use, database
driver in
> use, an
> MCVE code example that we can run (see
> http://stackoverflow.com/help/mcve
<http://stackoverflow.com/help/mcve>
> <http://stackoverflow.com/help/mcve
<http://stackoverflow.com/help/mcve>>).
>
>
>
>
> >
> > --
> > You received this message because you are subscribed to the
> Google
> > Groups "sqlalchemy" group.
> > To unsubscribe from this group and stop receiving emails
from
> it, send
> > an email to sqlalchemy+...@googlegroups.com
> > <mailto:sqlalchemy+unsubscr...@googlegroups.com
<javascript:>>.
> > To post to this group, send email to
sqlal...@googlegroups.com
> > <mailto:sqlal...@googlegroups.com>.
> > Visit this group at
https://groups.google.com/group/sqlalchemy
<https://groups.google.com/group/sqlalchemy>
> <https://groups.google.com/group/sqlalchemy
<https://groups.google.com/group/sqlalchemy>>.
> > For more options, visit
https://groups.google.com/d/optout <https://groups.google.com/d/optout>
> <https://groups.google.com/d/optout
<https://groups.google.com/d/optout>>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it,
send
> an email to sqlalchemy+...@googlegroups.com <javascript:>
> <mailto:sqlalchemy+unsubscr...@googlegroups.com <javascript:>>.
> To post to this group, send email to sqlal...@googlegroups.com
<javascript:>
> <mailto:sqlal...@googlegroups.com <javascript:>>.
> Visit this group at https://groups.google.com/group/sqlalchemy
<https://groups.google.com/group/sqlalchemy>.
> For more options, visit https://groups.google.com/d/optout
<https://groups.google.com/d/optout>.
--
You received this message because you are subscribed to the Google
Groups "sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to sqlalchemy+unsubscr...@googlegroups.com
<mailto:sqlalchemy+unsubscr...@googlegroups.com>.
To post to this group, send email to sqlalchemy@googlegroups.com
<mailto:sqlalchemy@googlegroups.com>.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
