[sqlalchemy] Re: Ascii codec instead of unicode ?
Sounds good. Thanks Jason!
On 8/20/07, jason kirtland [EMAIL PROTECTED] wrote:
Arun wrote:
So in short if I specify use_unicode=True at the SA engine level
then I can skip specifying use_unicode and specify only
charset=utf8 at mysqldb level ?
If you configure this DB-API driver for all-Unicode (which is what
happens when you only give it a 'charset') all strings will come
back from the database to SQLAlchemy as Unicode. You can ask the
Engine and/or types to convert_unicode=True, but it won't do
anything except add processing overhead- the strings are already
Unicode from the driver.
Try playing with the following to find a combination that suits
your needs. The first two engine configurations aren't options for
you obviously, but they make a good demo.
from sqlalchemy import *
e = create_engine('mysql:///test')
#e = create_engine('mysql:///test', convert_unicode=True)
#e = create_engine('mysql:///test?charset=utf8')
#e = create_engine('mysql:///test?charset=utf8',
# convert_unicode=True)
#e = create_engine('mysql:///test?charset=utf8use_unicode=0')
#e = create_engine('mysql:///test?charset=utf8use_unicode=0',
# convert_unicode=True)
m = MetaData(e)
t = Table('unicodings', m,
Column('string', String(32)),
Column('unicode', Unicode(32)))
if not t.exists():
t.create()
t.insert().execute({'string':'foo',
'unicode':'bar'})
print repr(list(t.select().execute()))
--
Cheers,
- A
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To post to this group, send email to sqlalchemy@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en
-~--~~~~--~~--~--~---
[sqlalchemy] Re: Ascii codec instead of unicode ?
So in short if I specify use_unicode=True at the SA engine level then I can skip specifying use_unicode and specify only charset=utf8 at mysqldb level ? On 8/19/07, jason kirtland [EMAIL PROTECTED] wrote: Arun Kumar PG wrote: Ok, you need to get that charset to the driver. Try removing SET NAMES from your init_command, and instead pass charset=utf8 and use_unicode=0 in your database connection URL. why do we want to say use_unicode=0 instead or use_unicode=True here? You can go either way with that. The MySQLdb driver's default behavior when given a 'charset' is to also turn on its return all strings in Unicode mode. If you want all of your strings as Unicode that's just dandy, but if you expecting them to come back as regular strings encoded in the charset you requested you'd be in for a surprise... In my own code I enable use_unicode and I don't specify any Unicode options or column types at the SQLAlchemy level. -j -- Cheers, - A --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Ascii codec instead of unicode ?
Arun wrote:
So in short if I specify use_unicode=True at the SA engine level
then I can skip specifying use_unicode and specify only
charset=utf8 at mysqldb level ?
If you configure this DB-API driver for all-Unicode (which is what
happens when you only give it a 'charset') all strings will come
back from the database to SQLAlchemy as Unicode. You can ask the
Engine and/or types to convert_unicode=True, but it won't do
anything except add processing overhead- the strings are already
Unicode from the driver.
Try playing with the following to find a combination that suits
your needs. The first two engine configurations aren't options for
you obviously, but they make a good demo.
from sqlalchemy import *
e = create_engine('mysql:///test')
#e = create_engine('mysql:///test', convert_unicode=True)
#e = create_engine('mysql:///test?charset=utf8')
#e = create_engine('mysql:///test?charset=utf8',
# convert_unicode=True)
#e = create_engine('mysql:///test?charset=utf8use_unicode=0')
#e = create_engine('mysql:///test?charset=utf8use_unicode=0',
# convert_unicode=True)
m = MetaData(e)
t = Table('unicodings', m,
Column('string', String(32)),
Column('unicode', Unicode(32)))
if not t.exists():
t.create()
t.insert().execute({'string':'foo',
'unicode':'bar'})
print repr(list(t.select().execute()))
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To post to this group, send email to sqlalchemy@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en
-~--~~~~--~~--~--~---
[sqlalchemy] Re: Ascii codec instead of unicode ?
why do we want to say use_unicode=0 instead or use_unicode=True here? On 8/16/07, jason kirtland [EMAIL PROTECTED] wrote: Ok you need to get tArun wrote: I am using mysqldb-1.2.2. I am passing 'SET NAMES' to connect method as a value for init_command parameter. All tables have utf8 charset. And I pass convert_unicode=True to engine. Let me know if anything else is required. Ok, you need to get that charset to the driver. Try removing SET NAMES from your init_command, and instead pass charset=utf8 and use_unicode=0 in your database connection URL. -- Cheers, - A --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Ascii codec instead of unicode ?
Arun Kumar PG wrote: Ok, you need to get that charset to the driver. Try removing SET NAMES from your init_command, and instead pass charset=utf8 and use_unicode=0 in your database connection URL. why do we want to say use_unicode=0 instead or use_unicode=True here? You can go either way with that. The MySQLdb driver's default behavior when given a 'charset' is to also turn on its return all strings in Unicode mode. If you want all of your strings as Unicode that's just dandy, but if you expecting them to come back as regular strings encoded in the charset you requested you'd be in for a surprise... In my own code I enable use_unicode and I don't specify any Unicode options or column types at the SQLAlchemy level. -j --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Ascii codec instead of unicode ?
Whooops. the problem in coming on the box which is running mysqldb1.2.0. Actually it's a non-upgraded server with mysqldb 1.2.0. I was looking at the code on my box which contains 1.2.2. But I am sure that even with mysqldb 1.2.0 the existing SA version was working fine. and that's why init_command is being used there as charset is not there in 1.2.0 as connect kwargs. Any clue? On 8/16/07, jason kirtland [EMAIL PROTECTED] wrote: Ok you need to get tArun wrote: I am using mysqldb-1.2.2. I am passing 'SET NAMES' to connect method as a value for init_command parameter. All tables have utf8 charset. And I pass convert_unicode=True to engine. Let me know if anything else is required. Ok, you need to get that charset to the driver. Try removing SET NAMES from your init_command, and instead pass charset=utf8 and use_unicode=0 in your database connection URL. -- Cheers, - A --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Ascii codec instead of unicode ?
Arun Kumar PG wrote: Hi All, Recently I upgraded to the version 3.9 of SA. Post that whenever I am trying to save characters in different language in the table I am getting the below exception: File /src/sqlalchemy/engine/base.py, line 601, in _execute raise exceptions.SQLError(context.statement, context.parameters, e) SQLError: (UnicodeDecodeError) 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128) I am wondering why it is using ascii codec instead of unicode ? FYI: I am using MySQL 4.1 and the charset of table is utf-8. Odd to see ascii there instead of latin1. Is your database configured for utf-8 client connections? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Ascii codec instead of unicode ?
All tables are having a charset of utf8. Additionally, I am issuing SET NAMES 'utf8' statement as a part of connection establishment. Anything that is wrong here or missing ? On 8/15/07, jason kirtland [EMAIL PROTECTED] wrote: Arun Kumar PG wrote: Hi All, Recently I upgraded to the version 3.9 of SA. Post that whenever I am trying to save characters in different language in the table I am getting the below exception: File /src/sqlalchemy/engine/base.py, line 601, in _execute raise exceptions.SQLError(context.statement, context.parameters, e) SQLError: (UnicodeDecodeError) 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128) I am wondering why it is using ascii codec instead of unicode ? FYI: I am using MySQL 4.1 and the charset of table is utf-8. Odd to see ascii there instead of latin1. Is your database configured for utf-8 client connections? -- Cheers, - A --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Ascii codec instead of unicode ?
Arun Kumar PG wrote:
All tables are having a charset of utf8. Additionally, I am issuing SET
NAMES 'utf8' statement as a part of connection establishment.
Anything that is wrong here or missing ?
Are you updating the connection character set in the driver as well
after issuing SET NAMES? It sounds like you might be out of sync and
the driver is still trying to convert unicode with its default character
set.
It'd be something like dbapi_con.set_character_set('utf8'), and I'm
pretty sure that will also issue a basic SET NAMES for you behind the
scenes.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To post to this group, send email to sqlalchemy@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en
-~--~~~~--~~--~--~---
[sqlalchemy] Re: Ascii codec instead of unicode ?
are you using convert_unicode=True and/or the Unicode type ? On Aug 15, 2007, at 12:01 PM, Arun Kumar PG wrote: All tables are having a charset of utf8. Additionally, I am issuing SET NAMES 'utf8' statement as a part of connection establishment. Anything that is wrong here or missing ? On 8/15/07, jason kirtland [EMAIL PROTECTED] wrote: Arun Kumar PG wrote: Hi All, Recently I upgraded to the version 3.9 of SA. Post that whenever I am trying to save characters in different language in the table I am getting the below exception: File /src/sqlalchemy/engine/base.py, line 601, in _execute raise exceptions.SQLError(context.statement, context.parameters, e) SQLError: (UnicodeDecodeError) 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128) I am wondering why it is using ascii codec instead of unicode ? FYI: I am using MySQL 4.1 and the charset of table is utf-8. Odd to see ascii there instead of latin1. Is your database configured for utf-8 client connections? -- Cheers, - A --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Ascii codec instead of unicode ?
Yes. it's being done. I create the engine and then set convert unicode = True. On 8/15/07, Michael Bayer [EMAIL PROTECTED] wrote: are you using convert_unicode=True and/or the Unicode type ? On Aug 15, 2007, at 12:01 PM, Arun Kumar PG wrote: All tables are having a charset of utf8. Additionally, I am issuing SET NAMES 'utf8' statement as a part of connection establishment. Anything that is wrong here or missing ? On 8/15/07, jason kirtland [EMAIL PROTECTED] wrote: Arun Kumar PG wrote: Hi All, Recently I upgraded to the version 3.9 of SA. Post that whenever I am trying to save characters in different language in the table I am getting the below exception: File /src/sqlalchemy/engine/base.py, line 601, in _execute raise exceptions.SQLError(context.statement, context.parameters, e) SQLError: (UnicodeDecodeError) 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128) I am wondering why it is using ascii codec instead of unicode ? FYI: I am using MySQL 4.1 and the charset of table is utf-8. Odd to see ascii there instead of latin1. Is your database configured for utf-8 client connections? -- Cheers, - A -- Cheers, - A --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Ascii codec instead of unicode ?
Any other clue that may be helpful in troubleshooting the cause ? On 8/15/07, Arun Kumar PG [EMAIL PROTECTED] wrote: Yes. it's being done. I create the engine and then set convert unicode = True. On 8/15/07, Michael Bayer [EMAIL PROTECTED] wrote: are you using convert_unicode=True and/or the Unicode type ? On Aug 15, 2007, at 12:01 PM, Arun Kumar PG wrote: All tables are having a charset of utf8. Additionally, I am issuing SET NAMES 'utf8' statement as a part of connection establishment. Anything that is wrong here or missing ? On 8/15/07, jason kirtland [EMAIL PROTECTED] wrote: Arun Kumar PG wrote: Hi All, Recently I upgraded to the version 3.9 of SA. Post that whenever I am trying to save characters in different language in the table I am getting the below exception: File /src/sqlalchemy/engine/base.py, line 601, in _execute raise exceptions.SQLError(context.statement, context.parameters, e) SQLError: (UnicodeDecodeError) 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128) I am wondering why it is using ascii codec instead of unicode ? FYI: I am using MySQL 4.1 and the charset of table is utf-8. Odd to see ascii there instead of latin1. Is your database configured for utf-8 client connections? -- Cheers, - A -- Cheers, - A -- Cheers, - A --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Ascii codec instead of unicode ?
Hi, I have two questions: 1) what exact versions of MySQL (i.e. 4.1.22) and py-mysqldb? 2) how many databases and/or tables (an encodings) used in your application? On Aug 15, 9:57 pm, Arun Kumar PG [EMAIL PROTECTED] wrote: Any other clue that may be helpful in troubleshooting the cause ? On 8/15/07, Arun Kumar PG [EMAIL PROTECTED] wrote: Yes. it's being done. I create the engine and then set convert unicode = True. On 8/15/07, Michael Bayer [EMAIL PROTECTED] wrote: are you using convert_unicode=True and/or the Unicode type ? On Aug 15, 2007, at 12:01 PM, Arun Kumar PG wrote: All tables are having a charset of utf8. Additionally, I am issuing SET NAMES 'utf8' statement as a part of connection establishment. Anything that is wrong here or missing ? On 8/15/07, jason kirtland [EMAIL PROTECTED] wrote: Arun Kumar PG wrote: Hi All, Recently I upgraded to the version 3.9 of SA. Post that whenever I am trying to save characters in different language in the table I am getting the below exception: File /src/sqlalchemy/engine/base.py, line 601, in _execute raise exceptions.SQLError(context.statement, context.parameters, e) SQLError: (UnicodeDecodeError) 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128) I am wondering why it is using ascii codec instead of unicode ? FYI: I am using MySQL 4.1 and the charset of table is utf-8. Odd to see ascii there instead of latin1. Is your database configured for utf-8 client connections? -- Cheers, - A -- Cheers, - A -- Cheers, - A --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Ascii codec instead of unicode ?
So to recap, you are setting the character set on the dbapi connection via the MySQLdb method in addition to issuing a manual SET NAMES query? Arun wrote: Any other clue that may be helpful in troubleshooting the cause ? On 8/15/07, Arun Kumar PG [EMAIL PROTECTED] wrote: Yes. it's being done. I create the engine and then set convert unicode = True. On 8/15/07, Michael Bayer [EMAIL PROTECTED] wrote: are you using convert_unicode=True and/or the Unicode type ? On Aug 15, 2007, at 12:01 PM, Arun Kumar PG wrote: All tables are having a charset of utf8. Additionally, I am issuing SET NAMES 'utf8' statement as a part of connection establishment. Anything that is wrong here or missing ? On 8/15/07, jason kirtland [EMAIL PROTECTED] wrote: Arun Kumar PG wrote: Hi All, Recently I upgraded to the version 3.9 of SA. Post that whenever I am trying to save characters in different language in the table I am getting the below exception: File /src/sqlalchemy/engine/base.py, line 601, in _execute raise exceptions.SQLError(context.statement, context.parameters, e) SQLError: (UnicodeDecodeError) 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128) I am wondering why it is using ascii codec instead of unicode ? FYI: I am using MySQL 4.1 and the charset of table is utf-8. Odd to see ascii there instead of latin1. Is your database configured for utf-8 client connections? -- Cheers, - A -- Cheers, - A --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Ascii codec instead of unicode ?
I am using mysqldb-1.2.2. I am passing 'SET NAMES' to connect method as a value for init_command parameter. All tables have utf8 charset. And I pass convert_unicode=True to engine. Let me know if anything else is required. thanks! On 8/15/07, jason kirtland [EMAIL PROTECTED] wrote: So to recap, you are setting the character set on the dbapi connection via the MySQLdb method in addition to issuing a manual SET NAMES query? Arun wrote: Any other clue that may be helpful in troubleshooting the cause ? On 8/15/07, Arun Kumar PG [EMAIL PROTECTED] wrote: Yes. it's being done. I create the engine and then set convert unicode = True. On 8/15/07, Michael Bayer [EMAIL PROTECTED] wrote: are you using convert_unicode=True and/or the Unicode type ? On Aug 15, 2007, at 12:01 PM, Arun Kumar PG wrote: All tables are having a charset of utf8. Additionally, I am issuing SET NAMES 'utf8' statement as a part of connection establishment. Anything that is wrong here or missing ? On 8/15/07, jason kirtland [EMAIL PROTECTED] wrote: Arun Kumar PG wrote: Hi All, Recently I upgraded to the version 3.9 of SA. Post that whenever I am trying to save characters in different language in the table I am getting the below exception: File /src/sqlalchemy/engine/base.py, line 601, in _execute raise exceptions.SQLError(context.statement, context.parameters, e) SQLError: (UnicodeDecodeError) 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128) I am wondering why it is using ascii codec instead of unicode ? FYI: I am using MySQL 4.1 and the charset of table is utf-8. Odd to see ascii there instead of latin1. Is your database configured for utf-8 client connections? -- Cheers, - A -- Cheers, - A -- Cheers, - A --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Ascii codec instead of unicode ?
Ok you need to get tArun wrote: I am using mysqldb-1.2.2. I am passing 'SET NAMES' to connect method as a value for init_command parameter. All tables have utf8 charset. And I pass convert_unicode=True to engine. Let me know if anything else is required. Ok, you need to get that charset to the driver. Try removing SET NAMES from your init_command, and instead pass charset=utf8 and use_unicode=0 in your database connection URL. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
