[sqlalchemy] Re: Conventions for creating SQL alchemy apps
some may-be-stupid answers: - see the number of lines per file - split it into app-field-related parts, not SA-arhitectural parts - hell, do as it is easier - start as one file, once u hit some limit of your nerve(r)s, split.. but do keep one single file as main entrance point On Thursday 20 December 2007 09:37:26 Morgan wrote: Hi Guys, This may be a stupid question so flame away I don't care, but I have been wondering. Is there a better way to layout your SQL alchemy files that my random method? Does anyone have a convention that works well for them. I'm only asking this because I cannot decide how I want to lay out the SQLAlchemy component of my application. I'm thinking of putting it all in files like engines.py, mapping.py, metadata.py etc or should I just shove this all in one file. Let me know if I have had too much coffee or not. Morgan --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: 0.4 deprecation warnings, what is the new get_by/select_by?
On Dec 20, 2007 8:50 AM, iain duncan [EMAIL PROTECTED] wrote: Sorry if this seems a stupid question, but I thought that Mike had said that in sa0.4, if you used session_context that this User.query.get_by(name='john') was the replacement for the old assign mapper convenience call. But I'm getting deprecation warnings. What should I be doing instead of the (almost as) convenient: User.query.get_by( **kwargs ) User.query.filter_by(**kwargs).first() User.query.select_by( **kwargs ) User.query.filter_by(**kwargs).all() User.query.select() User.query.filter(xxx).all() -- Gaëtan de Menten http://openhex.org --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Caching
Mike, thanks for your reply.
what happens if you just leave _state alone ? there shouldnt be any
need to mess with _state (nor _entity_name). the only attribute
worth deleting for the cache operation is _sa_session_id so that the
instance isnt associated with any particular session when it gets
cached. Id also consider using session.merge(dont_load=True) which is
designed for use with caches (and also watch out for that log.debug(),
debug() calls using the standard logging module are notoriously slow).
The reason for deleting _state is to save some space in cache. I save
instances to cache on get operation, so they are unmodified. But, of
course, it is internal thing so the final decision is yours :)
I gave up trying merge(dont_load=True) after running this sample:
users = Table('users', metadata,
Column('id', Integer, primary_key=True),
Column('name', String(100)),
Column('surname', String(100)))
mapper(User, users,
properties={
'surname': deferred(users.c.surname)
})
s = create_session()
u = User()
u.name = 'anton'
u.surname = 'belyaev'
s.save(u)
s.flush()
# now we need an instance with not loaded surname (because it is
deferred)
s = create_session()
u = s.query(User).get(1)
# cache it
cache = pickle.dumps(u)
# try to restore in a new session
s = create_session()
u = pickle.loads(cache)
u = s.merge(u, dont_load=True)
The latest statement fails with:
File /home/anton/eggs/lib/python2.5/site-packages/SQLAlchemy-0.4.1-
py2.5.egg/sqlalchemy/orm/session.py, line 1136, in object_session
if obj in sess:
TypeError: argument of type 'NoneType' is not iterable
Some notes on this test case:
1) If surname was a simple (not deferred) column property, merge
would work fine.
2) session.update(u) instead of merge would work fine even with
deferred column property, and the property itself would work fine (it
would load on first reference).
the only trac ticket for this is #490, which with our current
extension architecture is pretty easy to fix so its resolved in 3967 -
MapperExtensions are now fully inherited. If you apply the same
MapperExtension explicitly to a base mapper and a subclass mapper,
using the same ME instance will have the effect of it being applied
only once (and using two different ME instances will have the effect
of both being applied to the subclass separately).
I meant #870 (sorry, I should had provided reference in the first
message).
Back again to the testcase and note 2:
If, let say, I had some inheritance:
class Teacher(User):
pass
with polymorphic_fetch='deferred' (this is important), even
session.update(u) would not work. Because in this case deferred
attributes work through callables in _state, and callable does not
survive pickling.
Thanks.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To post to this group, send email to sqlalchemy@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en
-~--~~~~--~~--~--~---
[sqlalchemy] query.filter_or() ?
query.filter() does criterion = criterion new why not having one that does criterion = criterion | new ? its useful to have some query.this.that.filter.whatever.filter_or(...) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: query.filter_or() ?
How would you specify the logical parenthesis to control evaluation order in complex expressions? On Dec 20, 2007 5:14 AM, svilen [EMAIL PROTECTED] wrote: query.filter() does criterion = criterion new why not having one that does criterion = criterion | new ? its useful to have some query.this.that.filter.whatever.filter_or(...) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: 0.4 deprecation warnings, what is the new get_by/select_by?
On Thu, 2007-20-12 at 09:58 +0100, Gaetan de Menten wrote: On Dec 20, 2007 8:50 AM, iain duncan [EMAIL PROTECTED] wrote: Sorry if this seems a stupid question, but I thought that Mike had said that in sa0.4, if you used session_context that this User.query.get_by(name='john') was the replacement for the old assign mapper convenience call. But I'm getting deprecation warnings. What should I be doing instead of the (almost as) convenient: User.query.get_by( **kwargs ) User.query.filter_by(**kwargs).first() User.query.select_by( **kwargs ) User.query.filter_by(**kwargs).all() User.query.select() User.query.filter(xxx).all() Thanks, that makes sense now. Iain --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Caching
pickle isnt going to work with deferred columns unless you implement __getstate__ and __setstate__. so the issue with session.merge() is just an extension of that issue, correct ? i.e. without deferreds merge has no issue. is it not reasonable to ask that objects which are to be serialized and cached not have any deferred columns ? (or they are explicitly loaded before caching )? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Conventions for creating SQL alchemy apps
Agreed, it's easier to start with a single file, and then split from there as maintaining it becomes more difficult. You'll find the table defs and mappers, and mapped classes are pretty inter-related, and you'll drive yourself nuts if you have them in 20 different places. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Caching
pickle isnt going to work with deferred columns unless you implement __getstate__ and __setstate__. so the issue with session.merge() is just an extension of that issue, correct ? i.e. without deferreds merge has no issue. is it not reasonable to ask that objects which are to be serialized and cached not have any deferred columns ? (or they are explicitly loaded before caching )? Sorry, I dont understand clearly. I do understand that pickle saves only __dict__ when no __getstate__ defined. So, to be cached, an object should fetch all its deferred columns (if any) and provide all of them at __getstate__. Right? And if an instance from cache has nothing for one of its deferred column values, then referencing these properties after merge wont load them from DB, but just fail? Thanks. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Caching
On Dec 20, 2007, at 1:04 PM, Anton V. Belyaev wrote: So, to be cached, an object should fetch all its deferred columns (if any) and provide all of them at __getstate__. Right? that would work. And if an instance from cache has nothing for one of its deferred column values, then referencing these properties after merge wont load them from DB, but just fail? as far as merge failing, I need to see what the exact mechanics of that error message are. For a polymorphic deferred in particular, its a major chunk of an object's state that is deferred, i.e. everything corresponding to the joined tables, and the callables are currently established at the per-instance level. So it may be necessary now for merge to still fail if unloadable deferreds are detected, although we can and should provide a nicer error message. Some longer term solutions to the pickled issue include trying to be more aggressive about placing class-level attribute loaders which dont need to be serialized, placing hints in the _state which could help the _state to reconstruct the per-instance deferred callables, or we might even be able to get the _state to call deferreds during serialization without the need for an explicit __getstate__, but then you are caching all that additional state. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Conventions for creating SQL alchemy apps
Thanks for that, make sense to me. svilen wrote: some may-be-stupid answers: - see the number of lines per file - split it into app-field-related parts, not SA-arhitectural parts - hell, do as it is easier - start as one file, once u hit some limit of your nerve(r)s, split.. but do keep one single file as main entrance point On Thursday 20 December 2007 09:37:26 Morgan wrote: Hi Guys, This may be a stupid question so flame away I don't care, but I have been wondering. Is there a better way to layout your SQL alchemy files that my random method? Does anyone have a convention that works well for them. I'm only asking this because I cannot decide how I want to lay out the SQLAlchemy component of my application. I'm thinking of putting it all in files like engines.py, mapping.py, metadata.py etc or should I just shove this all in one file. Let me know if I have had too much coffee or not. Morgan --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] sqlalchemy object serialization / deserialization
Hi all,
I'm trying to implement simple object serialization (ala the pickle
module) using JSON. I'm pretty far along, but having one problem --
serializing a sqlalchemy object through the __reduce__ method produces
a circular reference through the InstanceState object.
pprint of the structures in question:
account.__reduce__():
(function _reconstructor at 0x42530,
(class 'model.account.Account',
type 'object',
None),
{'_comment': None,
'_description': None,
'_display_name': u'm',
'_email': u'[EMAIL PROTECTED]',
'_email_status': 1L,
'_entity_name': None,
'_instance_key': (class 'model.account.Account',
(5L,),
None),
'_name_graphic': u'm',
'_password_hash': u'bar',
'_sa_session_id': 43005744,
'_state': sqlalchemy.orm.attributes.InstanceState object at
0x286f978, InstanceState object
'_status': 1L,
'_uri': None,
'_user_name': u'm',
'account_id': 5L,
'avatar_exists': False,
'creation_time': datetime.datetime(2006, 8, 23, 17, 43, 26),
'modification_time': datetime.datetime(2007, 12, 19, 23, 3, 2),
'newsletter': False,
'reviewed': False,
'site_updates': False})
account._state.__reduce__():
(function _reconstructor at 0x42530,
(class 'sqlalchemy.orm.attributes.InstanceState', type 'object',
None),
{'committed_state': {'_comment': None,
'_description': None,
'_display_name': u'm',
'_email': u'[EMAIL PROTECTED]',
'_email_status': 1L,
'_name_graphic': u'm',
'_password_hash': u'bar',
'_status': 1L,
'_uri': None,
'_user_name': u'm',
'account_id': 5L,
'avatar_exists': False,
'creation_time': datetime.datetime(2006, 8, 23,
17, 43, 26),
'modification_time': datetime.datetime(2007, 12,
19, 23, 3, 2),
'newsletter': False,
'reviewed': False,
'site_updates': False},
'instance': model.account.Account object at 0x2971350,---
reference back to Account object
'modified': False,
'parents': {}})
So right now, I just break the circular reference the second time I
see the Account object, but this causes problems deserializing since
the InstanceState object is missing some data...
Any thoughts appreciated here... pickle is able to handle this
structure fine, but I'm not sure exactly what algorithm it uses to do
this...
m
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To post to this group, send email to sqlalchemy@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en
-~--~~~~--~~--~--~---
[sqlalchemy] Re: sqlalchemy object serialization / deserialization
You're not going to be able to serialize Python class instances in JSON: json strings are simple object literals limited to basic Javascript types. Pickle does some pretty heavy lifting to serialize and reconstitute class instances. Easiest way to store JSON in the database is to limit the type of data you store in JSON strings to straightforward objects that only use primitive JS types, and then serialize back and forth to Python dictionaries. That's what libraries like SimpleJSON or cJSON do. Using Sqlalchemy, you can then store those JSON strings in database VARCHARS, TEXT and so on fields. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: 0.4 deprecation warnings, what is the new get_by/select_by?
Maybe this could go into the FAQ? This post really helped me update some code using SA 0.3. Gaetan de Menten wrote: On Dec 20, 2007 8:50 AM, iain duncan [EMAIL PROTECTED] wrote: Sorry if this seems a stupid question, but I thought that Mike had said that in sa0.4, if you used session_context that this User.query.get_by(name='john') was the replacement for the old assign mapper convenience call. But I'm getting deprecation warnings. What should I be doing instead of the (almost as) convenient: User.query.get_by( **kwargs ) User.query.filter_by(**kwargs).first() User.query.select_by( **kwargs ) User.query.filter_by(**kwargs).all() User.query.select() User.query.filter(xxx).all() --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: sqlalchemy object serialization / deserialization
On Dec 20, 5:04 pm, Rick Morrison [EMAIL PROTECTED] wrote:
You're not going to be able to serialize Python class instances in JSON:
json strings are simple object literals limited to basic Javascript types.
Pickle does some pretty heavy lifting to serialize and reconstitute class
instances.
I've already figured this out -- you can use class hinting as
outlined in this page:
http://json-rpc.org/wiki/specification
So a datetime using this format and converted to JSON might look like:
modification_time: {__jsonclass__: [datetime.datetime, [2007,
12, 19, 23, 3, 2, 2]]}
My deserialization routine loads the datetime module, then gets the
datetime attribute (which in this case is a type, but could be a
function too), and calls that with the arguments in the list. This
sort of thing works for generic classes too using the pickle
__reduce__ hooks. The JSON part I'm actually handling through cjson
or simplejson -- my serialization/deserialization is working all on
python objects.
Easiest way to store JSON in the database is to limit the type of data you
store in JSON strings to straightforward objects that only use primitive JS
types, and then serialize back and forth to Python dictionaries. That's what
libraries like SimpleJSON or cJSON do. Using Sqlalchemy, you can then store
those JSON strings in database VARCHARS, TEXT and so on fields.
I don't really want to store JSON in the DB, but just use it as a
serialization format for sqlalchemy objects. We want our frontend to
render data from the same type of object with a couple different
possible backend sources. One being the database through the
sqlalchemy ORM and another being some sort of JSON/XML interface that
we can backend from whatever...
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To post to this group, send email to sqlalchemy@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en
-~--~~~~--~~--~--~---
[sqlalchemy] Re: sqlalchemy object serialization / deserialization
Hey Matt.
Class hinting was a json-rpc 1.0 feature, subsequently dropped in 1.1. But
that's a nit.
The real problem, though is reconstituing those hints -- the machinery for
reinstantiating those objects has to come from somewhere, and that somewhere
is going to your code -- there is no magic just add water and reconstitute
my object built into Python.
If you really want to get into writing your own deserialization, you're
already heading in the right direction: simplejson and cjson (and maybe
others) -- have hooks to plug in your own serialization and deserialization
code. I've used them both to do something almost exactly like this to
serialize and deserialize JS dates, which normally can't be sent in JSON.
You'll find that the built-in serialization works for *most* Python objects,
especially for basic types like str(), int(), etc., but most certainly not
*all* Python objects.
I don't really want to store JSON in the DB, but just use it as a
serialization format for sqlalchemy objects. We want our frontend to
render data from the same type of object with a couple different
possible backend sources. One being the database through the
sqlalchemy ORM and another being some sort of JSON/XML interface that
we can backend from whatever...
Your best bet is going to be to deserialize the JSON to some intermediate
object and then construct the SqlAlchemy instance. Re-implementing pickle
with a JSON storage format is going to be a huge job, and really won't give
you anything that you don't already get from pickle -- what other app
besides you own is ever going to understand those heavily class-hinted JSON
files?
On 12/20/07, Matt [EMAIL PROTECTED] wrote:
On Dec 20, 5:04pm, Rick Morrison [EMAIL PROTECTED] wrote:
You're not going to be able to serialize Python class instances in JSON:
json strings are simple object literals limited to basic Javascript
types.
Pickle does some pretty heavy lifting to serialize and reconstitute
class
instances.
I've already figured this out -- you can use class hinting as
outlined in this page:
http://json-rpc.org/wiki/specification
So a datetime using this format and converted to JSON might look like:
modification_time: {__jsonclass__: [datetime.datetime, [2007,
12, 19, 23, 3, 2, 2]]}
My deserialization routine loads the datetime module, then gets the
datetime attribute (which in this case is a type, but could be a
function too), and calls that with the arguments in the list. This
sort of thing works for generic classes too using the pickle
__reduce__ hooks. The JSON part I'm actually handling through cjson
or simplejson -- my serialization/deserialization is working all on
python objects.
Easiest way to store JSON in the database is to limit the type of data
you
store in JSON strings to straightforward objects that only use primitive
JS
types, and then serialize back and forth to Python dictionaries. That's
what
libraries like SimpleJSON or cJSON do. Using Sqlalchemy, you can then
store
those JSON strings in database VARCHARS, TEXT and so on fields.
I don't really want to store JSON in the DB, but just use it as a
serialization format for sqlalchemy objects. We want our frontend to
render data from the same type of object with a couple different
possible backend sources. One being the database through the
sqlalchemy ORM and another being some sort of JSON/XML interface that
we can backend from whatever...
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To post to this group, send email to sqlalchemy@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en
-~--~~~~--~~--~--~---
[sqlalchemy] Re: Caching
On Dec 20, 2007, at 1:04 PM, Anton V. Belyaev wrote: And if an instance from cache has nothing for one of its deferred column values, then referencing these properties after merge wont load them from DB, but just fail? I rearranged instance-level deferred loaders to be serializable instances in r3968. you can now pickle an instance + its _state and restore, and all deferred/lazy loaders will be restored as well. I didnt yet test it specifically with merge() but give it a try, you shoudnt be getting that error anymore...the pickling issue from ticket #870 is also no longer present. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
