[sqlalchemy] Re: mysql utf8 encoding problem
Michael Bayer wrote: On Oct 8, 2008, at 8:54 PM, jason kirtland wrote: Most likely you just need to configure the db-api's client encoding by adding ?charset=utf8 onto your connection URL. Enough folks have hit this recently that I'm (again) considering passing through the engine encoding= parameter to the MySQLdb connection setup. I've resisted the urge for a while because we don't to my knowledge re-configure any db-apis in any of the backends. But this keeps coming up despite being documented in the mysql section of the docs, and last time I traced through it, it seemed like MySQLdb was ignoring the server's configured connection_encoding so a little assist from the SA side would probably be useful. I'll look at sneaking that into the upcoming rc2 unless the implementation is untenable for some reason or there's an outcry. since im a total dumdum, why have i never had this issue in my own dealings with MySQL and Unicode ? I use the Unicode type, i dont use any charset= on my URL, and things work fine, including all of our unit tests. Is it actually storing the data incorrectly and we just see the same info at the SQLA round trip level ? i don't know about your env, but the unit tests under mysql use testing.engines.utf8_engine to configure a connection that can send unicode across the wire without encoding failures. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: mysql utf8 encoding problem
Michael Bayer wrote: On Oct 9, 2008, at 3:39 AM, jason kirtland wrote: i don't know about your env, but the unit tests under mysql use testing.engines.utf8_engine to configure a connection that can send unicode across the wire without encoding failures. I can run python test/sql/testtypes.py --db mysql --verbose UnicodeTest.testbasic with a raise or pdb.set_trace() inside of utf8_engine, and engines.utf8_engine is never called. Might be something about the data being tested in that test method. It's needed in the tests where it's used. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: mysql utf8 encoding problem
Thanks all! The connection string is getting unicode into the db intact. Is it possible that the charset is set in my.conf or elsewhere on some environments? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: mysql utf8 encoding problem
On Oct 9, 2008, at 3:39 AM, jason kirtland wrote: i don't know about your env, but the unit tests under mysql use testing.engines.utf8_engine to configure a connection that can send unicode across the wire without encoding failures. I can run python test/sql/testtypes.py --db mysql --verbose UnicodeTest.testbasic with a raise or pdb.set_trace() inside of utf8_engine, and engines.utf8_engine is never called. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: mysql utf8 encoding problem
On Oct 9, 2008, at 10:02 AM, Michael Bayer wrote: On Oct 9, 2008, at 3:39 AM, jason kirtland wrote: i don't know about your env, but the unit tests under mysql use testing.engines.utf8_engine to configure a connection that can send unicode across the wire without encoding failures. I can run python test/sql/testtypes.py --db mysql --verbose UnicodeTest.testbasic with a raise or pdb.set_trace() inside of utf8_engine, and engines.utf8_engine is never called. doh ! yupround trip works but data is stored as garbage. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: mysql utf8 encoding problem
I had similar problems, part of it turned out to be the encoding send
from the browser.
on forms you can set the encoding (in the formtag, check
http://www.w3schools.com
if the page is other than UTF-8 (check your browser view-
encoding menu).
I changed everything to utf-8 by setting headers and configuring the
webserver (apache in my case)
I'm not sure if you are developing web stuff though,
MySQLdb behaves odd when leaving the out the encoding though
so adding it into SA is a help which saves me from patching every SA
release as I do!
Martijn
On Oct 9, 2008, at 2:54 AM, jason kirtland wrote:
joelanman wrote:
Hi,
Firstly - I'm hugely impressed with SQLAlchemy - it's really helped
me
a lot with my new project.
I'm having problems storing international characters in mysql using
SQLAlchemy. For example:
école—school
looks like this in mysql:
école—school
I'm using the following engine call:
engine = create_engine(config.db, encoding='utf-8')
and using Unicode as the column type:
Column('content', UnicodeText),
and utf8 for the table:
mysql_charset='utf8'
I'm pretty sure all my mySQL options are set to utf8. This looks
really similar to a 'double encoding' issue I found while searching
the group, but it seems that was fixed in python-mysql 1.2.2, which
is
what I'm using.
Any help would be much appreciated.
Most likely you just need to configure the db-api's client encoding by
adding ?charset=utf8 onto your connection URL.
Enough folks have hit this recently that I'm (again) considering
passing
through the engine encoding= parameter to the MySQLdb connection
setup.
I've resisted the urge for a while because we don't to my knowledge
re-configure any db-apis in any of the backends. But this keeps
coming
up despite being documented in the mysql section of the docs, and last
time I traced through it, it seemed like MySQLdb was ignoring the
server's configured connection_encoding so a little assist from the SA
side would probably be useful.
I'll look at sneaking that into the upcoming rc2 unless the
implementation is untenable for some reason or there's an outcry.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To post to this group, send email to sqlalchemy@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en
-~--~~~~--~~--~--~---
[sqlalchemy] Re: mysql utf8 encoding problem
joelanman wrote:
Hi,
Firstly - I'm hugely impressed with SQLAlchemy - it's really helped me
a lot with my new project.
I'm having problems storing international characters in mysql using
SQLAlchemy. For example:
école—school
looks like this in mysql:
école—school
I'm using the following engine call:
engine = create_engine(config.db, encoding='utf-8')
and using Unicode as the column type:
Column('content', UnicodeText),
and utf8 for the table:
mysql_charset='utf8'
I'm pretty sure all my mySQL options are set to utf8. This looks
really similar to a 'double encoding' issue I found while searching
the group, but it seems that was fixed in python-mysql 1.2.2, which is
what I'm using.
Any help would be much appreciated.
Most likely you just need to configure the db-api's client encoding by
adding ?charset=utf8 onto your connection URL.
Enough folks have hit this recently that I'm (again) considering passing
through the engine encoding= parameter to the MySQLdb connection setup.
I've resisted the urge for a while because we don't to my knowledge
re-configure any db-apis in any of the backends. But this keeps coming
up despite being documented in the mysql section of the docs, and last
time I traced through it, it seemed like MySQLdb was ignoring the
server's configured connection_encoding so a little assist from the SA
side would probably be useful.
I'll look at sneaking that into the upcoming rc2 unless the
implementation is untenable for some reason or there's an outcry.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To post to this group, send email to sqlalchemy@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en
-~--~~~~--~~--~--~---
[sqlalchemy] Re: mysql utf8 encoding problem
On Oct 8, 2008, at 8:54 PM, jason kirtland wrote: Most likely you just need to configure the db-api's client encoding by adding ?charset=utf8 onto your connection URL. Enough folks have hit this recently that I'm (again) considering passing through the engine encoding= parameter to the MySQLdb connection setup. I've resisted the urge for a while because we don't to my knowledge re-configure any db-apis in any of the backends. But this keeps coming up despite being documented in the mysql section of the docs, and last time I traced through it, it seemed like MySQLdb was ignoring the server's configured connection_encoding so a little assist from the SA side would probably be useful. I'll look at sneaking that into the upcoming rc2 unless the implementation is untenable for some reason or there's an outcry. since im a total dumdum, why have i never had this issue in my own dealings with MySQL and Unicode ? I use the Unicode type, i dont use any charset= on my URL, and things work fine, including all of our unit tests. Is it actually storing the data incorrectly and we just see the same info at the SQLA round trip level ? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
