[sqlalchemy] Re: Can I coerce strings into Unicode?
Anybody?
On Jun 4, 1:13 am, AF allen.fow...@yahoo.com wrote:
Hello,
I'm using sqlite and convert_unicode = True on the engine.
How can I force coerce string based object attributes in to unicode?
(I had thought convert_unicode = True would do this)
Here is what I am seeing...
Setup code:
engine = create_engine('sqlite:///:memory:', echo=True,
convert_unicode=True)
Session = sessionmaker(bind=engine)
session = Session()
metadata = MetaData()
m1 = message(u'message body 1')
Now, in ipython:
In [1]: session.add(m1)
In [2]: m1.body
Out[2]: u'message body 1'
In [3]: m1.body = u'new - unicode'
In [4]: m1.body
Out[4]: u'new - unicode'
In [5]: m1.body = 'new - NOT unicode'
In [6]: m1.body
Out[6]: 'new - NOT unicode'
In [7]: unicode(m1.body)
Out[7]: u'new - NOT unicode'
Output line 6 is the problem.
Ideally, I'd like to see output lines 6 7 be the same.
Am I doing something wrong?
Thank you,
Allen
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To post to this group, send email to sqlalchemy@googlegroups.com
To unsubscribe from this group, send email to
sqlalchemy+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en
-~--~~~~--~~--~--~---
[sqlalchemy] Re: Can I coerce strings into Unicode?
The engine's conversion to unicode doesn't happen when you assign the
property, it happens when the underlying database operation is
committed, and arrives in the python object's property only after
roundtripping through the database.
In [50]: m1.body = 'new - NOT unicode'
In [51]: m1.body
Out[51]: 'new - NOT unicode'
In [52]: session.add(m1)
In [53]: m1.body
Out[53]: 'new - NOT unicode'
In [54]: session.commit()
2009-06-12 09:46:18,430 INFO sqlalchemy.engine.base.Engine.0x...aa70
BEGIN
2009-06-12 09:46:18,431 INFO sqlalchemy.engine.base.Engine.0x...aa70
INSERT INTO message (body) VALUES (?)
2009-06-12 09:46:18,431 INFO sqlalchemy.engine.base.Engine.0x...aa70
['new - NOT unicode']
2009-06-12 09:46:18,432 INFO sqlalchemy.engine.base.Engine.0x...aa70
COMMIT
In [55]: m1.body
2009-06-12 09:46:22,803 INFO sqlalchemy.engine.base.Engine.0x...aa70
BEGIN
2009-06-12 09:46:22,804 INFO sqlalchemy.engine.base.Engine.0x...aa70
SELECT message.body AS message_body
FROM message
WHERE message.body = ?
2009-06-12 09:46:22,805 INFO sqlalchemy.engine.base.Engine.0x...aa70
['new - NOT unicode']
Out[55]: u'new - NOT unicode'
So if you want to rely on the engine's coercion to unicode, you have
to go through the engine.
But really, you ought to take care of unicode encoding issues yourself
before passing text data to SQLAlchemy. The engine cannot know where
your 8-bit strings come from, and just assumes that they are encoded
in UTF-8 (or whatever other encoding you set it to). You are better
equipped to know what encoding to expect your string input to be in
(and it may vary, depending on your input). If you don't know, find
out. (Or specify and assert.)
Cheers,
- Gulli
On Jun 12, 6:20 am, allen.fowler allen.fow...@yahoo.com wrote:
Anybody?
On Jun 4, 1:13 am, AF allen.fow...@yahoo.com wrote:
Hello,
I'm using sqlite and convert_unicode = True on the engine.
How can I force coerce string based object attributes in to unicode?
(I had thought convert_unicode = True would do this)
Here is what I am seeing...
Setup code:
engine = create_engine('sqlite:///:memory:', echo=True,
convert_unicode=True)
Session = sessionmaker(bind=engine)
session = Session()
metadata = MetaData()
m1 = message(u'message body 1')
Now, in ipython:
In [1]: session.add(m1)
In [2]: m1.body
Out[2]: u'message body 1'
In [3]: m1.body = u'new - unicode'
In [4]: m1.body
Out[4]: u'new - unicode'
In [5]: m1.body = 'new - NOT unicode'
In [6]: m1.body
Out[6]: 'new - NOT unicode'
In [7]: unicode(m1.body)
Out[7]: u'new - NOT unicode'
Output line 6 is the problem.
Ideally, I'd like to see output lines 6 7 be the same.
Am I doing something wrong?
Thank you,
Allen
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To post to this group, send email to sqlalchemy@googlegroups.com
To unsubscribe from this group, send email to
sqlalchemy+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en
-~--~~~~--~~--~--~---
[sqlalchemy] Re: Can I coerce strings into Unicode?
On Jun 12, 6:00 am, Gunnlaugur Briem gunnlau...@gmail.com wrote: The engine's conversion to unicode doesn't happen when you assign the property, it happens when the underlying database operation is committed, and arrives in the python object's property only after roundtripping through the database. OK, i see. Are there any shortcuts for installing a filter on the object to mandate (and force if possible) UTF-8 on all incoming property assignments? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Can I coerce strings into Unicode?
allen.fowler wrote:
On Jun 12, 6:00 am, Gunnlaugur Briem gunnlau...@gmail.com wrote:
The engine's conversion to unicode doesn't happen when you assign the
property, it happens when the underlying database operation is
committed, and arrives in the python object's property only after
roundtripping through the database.
OK, i see.
Are there any shortcuts for installing a filter on the object to
mandate (and force if possible) UTF-8 on all incoming property
assignments?
I use this
class _coerce_from_utf8(TypeDecorator):
def process_bind_param(self, value, dialect):
if isinstance(value, str):
value = value.decode('utf-8')
return value
class UTF8(_coerce_from_utf8):
A Unicode type which coerces from utf-8.
impl = sa.Unicode
class UTF8Text(_coerce_from_utf8):
A Unicode type which coerces from utf-8.
impl = sa.UnicodeText
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To post to this group, send email to sqlalchemy@googlegroups.com
To unsubscribe from this group, send email to
sqlalchemy+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en
-~--~~~~--~~--~--~---
