Oh yeah, before I forget...
Regarding the object expiration: The behaviour as it is now in SA is
fine I guess, seeing as the purpose of having an ORM is having
instances reflect their status in the database at all times. No
session means no database and so the behaviour is going to be strange.
What I'm thinking of doing is using the class manager to set up a
separate dict for sql state, say obj.__sa_dict__. And thus on
expiration __sa_dict__ gets cleared instead of obj.__dict__ . The
trick will be in getting attribute access to behave correctly (I'm not
really sure what 'correct' is yet).
I'm thinking:
1. On initialization __dict__ should be copied to __sa_dict__
-> Easily done through the class mapper.
-> __dict__ is mostly empty on initialization though... ?
2. If there is an active object session values should be retrieved
via the object's state class, thus using __sa_dict__
-> Can be done through the object's __getattribute__ method...
3. When setting an attribute copy it to both dicts? Not sure about this yet...
4. On attribute expiration first copy the value to __dict__ so
everyone else still has access to it in case the session gets closed.
-> Makes a bit more sense then 3.
-> How to implement it though?
Thus once the session is closed we can continue working with the
instance and choose to either:
a. merge it back into a new session. Thus saving any changes we made
while the session was closed.
b. get a new instance from a new session via
sess.query(Class).get(id) and thus discard changes made in the mean
time.
Does this make sense? Is there an existing mechanism to do this already?
Or are there really important reasons never to work with instances
that aren't attached to any sessions? I'm thinking mainly that having
several copies of essentially the same data is not a good idea perhaps
and should be treated with care.
Some thoughts from anyone perhaps?
Y'all have a great week,
Christian
2009/3/22 Christiaan Putter <ceput...@googlemail.com>:
> Hi Svilen,
>
> Setting up tests is a very good idea, I'll start on some unit testing
> immediately. Definitely the best way to insure behaviour remains
> constant as you're working on the implementation, was just to lazy to
> do so up till now.
>
> Speaking of laziness, I've noticed that setting lazy=False on a
> relation does not effect refresh operations on the relation, it only
> loads on reconstruction. Having cascade='all' doesn't change this
> either. Is their some setting that I'm missing?
>
> Another little issue I'm trying to get fixed out is extending a
> class's attributes at run time, let's call it Parent. The need for
> this is allowing users to add custom fields (in a seperate table as
> the parent) at run time to Parent. My approach is letting the user
> set up a new class (ChildX), which inherits Base (not parent), with
> the attributes and methods that compute said attributes during run
> time. Then I compile that using exec, and and do some magic in its
> metaclass. Basically I add a relation to Parent with backref to
> ChildX and uselist=False on both sides. Enthought then allows me to
> add new traits to the Parent class that delegate their value the
> ChildX instance through the backref on Parent. This is basically just
> mapping the Parent class over several tables, with the ability to
> extend the class on the fly.
>
> It's mostly working. One problem though is that on reconstructing a
> Parent instance (or creating a new one) I have to have a mechanism
> that checks if all the Child classes have instances related to this
> record of Parent. Not a gig deal though.
>
> Is their a better way to do this? I'm doing some really ugly hacking
> to the Parent's mapper at run time, seeing as Mapper doesn't have a
> delete_property method.
>
> Should I try using normal inheritance instead? Will I be able to
> access ChildX.x from ChildY for instance? And what about changing
> ChildX's definition and running exec on it again?
>
> Perhaps someone has done something similar before and can give me some
> pointers.
>
> Thanks for all the advice up till now, it's been really useful.
>
> Have a great day,
> Christian
>
>
> 2009/3/22 <a...@svilendobrev.com>:
>>
>> good on yer!
>> one thing that i sorta failed is making a clean set of tests for this.
>> if u aren't too far in the usage, save some major future headaches and
>> do yourself a favour, make such test set. All the things u've tried
>> has to be (simple) testcases - it will be the _spec_ of how the thing
>> works / what u want from it. Anyone else on the same path could also
>> use it then ;-)
>>
>> i hope that once u do that u'll get an idea of where exactly u have
>> failed and fix it.
>>
>> as of expiration, it also did me some trouble. i have not stepped on
>> that session-gone one but i can see how it happens. The (expected)
>> life-times of SA-touched things aren't that obvious, and aren't
>> documented. e.g. Objects living longer than session or in an extreme
>> case, the mappers, might be troublesome.
>>
>> what i can suggest is, put debug statements and run with echo=True and
>> watch what goes on behind and how SA-activities intertwist with
>> traits' /your ones.
>>
>> ciao
>> svilen
>>
>> On Sunday 22 March 2009 04:00:27 Christiaan Putter wrote:
>>> Hi svilen,
>>>
>>> Thanks for your advice, going through your code helped a lot in
>>> understanding how SA works.
>>>
>>> I've gotten traited classes to behave like classes generated by
>>> SA's declarative extension. After a lot of stepping through code I
>>> realised the main problem was that SA removes values from an
>>> instance __dict__ once they have been expired. Traits smartly (or
>>> not so smartly) picks this up and on the next access to those
>>> attributes and returns their default values. Whereas SA would
>>> normally refresh the values from SQL. This caused some serious
>>> headaches, and really strange behaviour.
>>>
>>> My solution was to the overwrite the class's __getattribute__ with:
>>>
>>> def __getattribute__(self, key):
>>> try:
>>> dict = super(HasTraitsORM,
>>> self).__getattribute__('__dict__')
>>>
>>> if ('_sa_state' in dict):
>>> state = dict['_sa_state']
>>> if key in state.manager.keys():
>>> return state.get_impl(key).get(state)
>>>
>>> except Exception, e:
>>> print "Exception in SA getattr for %s: %s" % (key, e)
>>> pass
>>>
>>> return super(HasTraitsORM, self).__getattribute__(key)
>>>
>>>
>>> Which tries to first get an attribute from the instance's state
>>> (which will issue SQL if needed), and if that fails pass the
>>> request along to its super class where Traits will do it's magic if
>>> needed.
>>>
>>> This seems to work in the tests if been trying. Most of the work
>>> is done by SA declarative, and I've just added a thin layer on top
>>> to get it to behave well with Traited classes and map traits to SA
>>> columns.
>>>
>>> One issue I'm still having though is that after commits all
>>> attributes get expired. If you close the session the instance was
>>> attached to the data you just commited can't be accessed any more.
>>> This is also the behaviour of plain declarative classes. This is
>>> rather unintuitive I believe since once you've comitted an instance
>>> and closed the session you can't use the data you just set any
>>> more. In my case this means records being displayed in a gui get
>>> corrupted.
>>>
>>> Though what's strange is that if you access the attributes after
>>> the commit and before you close the session, and only then close
>>> the session, the values remain in the instance's __dict__ and the
>>> class behaves normally again. Is this intended for some reason?
>>> Is there a workaround? Personally I don't think quantum physics
>>> should apply to the bits of my programs and thus merely observing
>>> them should not change their behaviour. From going through the
>>> source I found a 'dont_expire_missing' attribute on attribute
>>> implementations, though this does unfortunately not do what I'd
>>> like.
>>>
>>> So what I'm doing now is keeping a session open on the gui's
>>> thread. Which causes some problems when trying to modify said
>>> instances from other threads and commiting the new values.
>>>
>>> How's this usually done is SA? Right now I've put a lock on the
>>> gui thread's session and allow other threads to issue a commit on
>>> that main thread if need be. I can't imagine this is the best way
>>> to do it...
>>>
>>> Hope you're all having a great weekend,
>>>
>>> Christian
>>>
>>> 2009/2/8 <a...@svilendobrev.com>:
>>> > afaiknow these traits are like my own static_types, i.e.
>>> > descriptors holding metadata and applying it to attribute access.
>>> > i have been combining SA with static_type for more than 2 years
>>> > now, since SA 3.0.
>>> >
>>> > The approach i did in the beginning was to replace the object's
>>> > __dict__ by something smart that is static_type-aware. When
>>> > InstrumentionManager framework came into place, i did not find it
>>> > any different, as SA still uses the __dict__ for data access. The
>>> > difference is that now my __dict__ replacement is created once
>>> > and not at every attr.access.
>>> >
>>> > i did suggest one patch about replaceing the SA's
>>> > obj.__dict__.whatever usage with an explicit set of methods (so
>>> > one knows what access to mime), and that was making SA 1-2%
>>> > faster, but it wasn't accepted.
>>> >
>>> > basicaly now there's a thin layer on top of SA, then SA itself,
>>> > then a thick layer underneath managing the data (the fake
>>> > __dict__).
>>> >
>>> > declarative+traits... u'll end up where i was. dbcook.sf.net is
>>> > doing that - since beginning. and it's switchable on/off.
>>> > It all works well and stable, in project with 250-300 classes,
>>> > although about 15% slower than without it (-:)
>>> >
>>> > The sa2static code:
>>> > svn co
>>> > http://dbcook.svn.sourceforge.net/svnroot/dbcook/trunk/dbcook/usa
>>> >ge/static_type/
>>> >
>>> > The static_type itself:
>>> > svn co
>>> > https://dbcook.svn.sourceforge.net/svnroot/dbcook/static_type
>>> >
>>> > whole dbcook:
>>> > svn co https://dbcook.svn.sourceforge.net/svnroot/dbcook/trunk
>>> >
>>> > i have quite some experience fighting this field, ask if u want.
>>> >
>>> > ciao
>>> > svilen
>>> > www.svilendobrev.com
>>> >
>>> > On Sunday 08 February 2009 01:51:20 cputter wrote:
>>> >> Hi guys and girls,
>>> >>
>>> >> I've recently discovered the joys of using sqlalchemy and would
>>> >> love to using it together with Traits. A few months back there
>>> >> was an attempt to integrate sqlalchemy into traits, though it
>>> >> wasn't really comprehensive in exploiting all of sqlalchemy's
>>> >> potential.
>>> >>
>>> >> So I'm trying to work on that and combine ext.Declarative with
>>> >> traits. The basic idea is to use the DeclarativeMeta type to
>>> >> generate Columns from Traits and pass those on for the
>>> >> Declarative extension to do its magic. This would allow mixing
>>> >> of sqlalchemy attributes and trait attributes in a single class
>>> >> so that we could still make use of all the relational setup
>>> >> sqlalchemy does in any case.
>>> >>
>>> >> Reading through several threads and looking at Elixir's SA
>>> >> integration helped me a bit though I couldn't find any
>>> >> documentation on how to implement the InstrumentationManager
>>> >> interface. I'm assuming this would be essential for letting
>>> >> Traits and SQLAlchemy play well together.
>>> >>
>>> >> There's still a lot of work to do, and I'm not really sure what
>>> >> needs to be done for everything to work properly. Would really
>>> >> appreciate it if someone could help me out.
>>> >>
>>> >>
>>> >> Here's an example of how it's working at the moment, I'll add
>>> >> the actual implementation at the end.
>>> >>
>>> >> #####################################
>>> >> class User(HasTraitsORM):
>>> >> __tablename__ = 'users'
>>> >>
>>> >> id = Column('id', Integer, primary_key=True)
>>> >> name = Str(sqldb=True)
>>> >>
>>> >> def _name_changed(self, old, new):
>>> >> print 'Changed name from %s to %s.' % (old, new)
>>> >>
>>> >> def __repr__(self):
>>> >> return '<User(%s, %s)>' % (self.id, self.name)
>>> >>
>>> >> people = ['John', 'Charls','Steve','Smith','Jane']
>>> >>
>>> >> for per in people:
>>> >> obj = User(name=per)
>>> >> sess = sqlservice.Session()
>>> >> sess.add(obj)
>>> >> sess.commit()
>>> >> sess.close()
>>> >> print obj
>>> >>
>>> >> session = sqlservice.Session()
>>> >> print '\nQuery all users\n'
>>> >> for user in session.query(User).order_by(User.name).all():
>>> >> print user
>>> >>
>>> >> session.close()
>>> >>
>>> >>
>>> >> ############################
>>> >>
>>> >> Which spits out:
>>> >> ###############################
>>> >> Changed name from to John.
>>> >> <User(users.id, )>
>>> >> Changed name from to Charls.
>>> >> <User(users.id, )>
>>> >> Changed name from to Steve.
>>> >> <User(users.id, Steve)>
>>> >> Changed name from to Smith.
>>> >> <User(users.id, Smith)>
>>> >> Changed name from to Jane.
>>> >> <User(users.id, Jane)>
>>> >>
>>> >> Query all users
>>> >>
>>> >> <User(2, Charls)>
>>> >> <User(1, John)>
>>> >>
>>> >> ##############################
>>> >>
>>> >> Which is really strange behaviour. There's obviously something
>>> >> wrong in my implementation of HasTraitsORM but why the different
>>> >> results within the same loop??? Why add only two instances?
>>> >>
>>> >> Totally baffles me.
>>> >>
>>> >> Here's the rest of my code, hope somehow can help me out. It's
>>> >> very messy, I've been hacking at it like crazy with no success
>>> >> :-)
>>> >>
>>> >> Hope you're all having a great weekend.
>>> >> -Chris
>>> >>
>>> >>
>>> >> ##############################
>>> >>
>>> >> # Standard library imports.
>>> >> import logging
>>> >>
>>> >> # Enthought library imports
>>> >> from enthought.preferences.api import Preferences
>>> >> from enthought.traits.api import \
>>> >> HasTraits, MetaHasTraits, Int, Str, Bool, Float, Any,\
>>> >> String, Enum, Python, \
>>> >> on_trait_change, TraitListObject
>>> >>
>>> >> # Package imports
>>> >> import sqlalchemy
>>> >> from sqlalchemy import Column, Integer
>>> >> from sqlalchemy.schema import MetaData
>>> >> from sqlalchemy.orm.interfaces import MapperProperty,
>>> >> InstrumentationManager
>>> >> from sqlalchemy.orm.attributes import get_attribute,
>>> >> set_attribute, is_instrumented
>>> >> from sqlalchemy.orm.collections import InstrumentedList,
>>> >> collection_adapter
>>> >> from sqlalchemy.ext.declarative import _as_declarative
>>> >>
>>> >>
>>> >> # Setup a logger for this module.
>>> >> logger = logging.getLogger(__name__)
>>> >>
>>> >>
>>> >> TRAIT_MAPPING = {
>>> >> Int : 'sqlalchemy.Integer',
>>> >> Str : 'sqlalchemy.Text',
>>> >> Enum : 'sqlalchemy.Text',
>>> >> String : 'sqlalchemy.Text',
>>> >> Float : 'sqlalchemy.Float',
>>> >> Bool : 'sqlalchemy.Boolean',
>>> >> }
>>> >>
>>> >>
>>> >> class HasTraitsORMState(InstrumentationManager):
>>> >> def __init__(self, cls):
>>> >> self.states = {}
>>> >>
>>> >> def instrument_attribute(self, class_, key, attr):
>>> >> pass
>>> >>
>>> >> def install_descriptor(self, class_, key, attr):
>>> >> pass
>>> >>
>>> >> def uninstall_descriptor(self, class_, key, attr):
>>> >> pass
>>> >>
>>> >> def instrument_collection_class(self, class_, key,
>>> >> collection_class):
>>> >> return ObjectCollection
>>> >>
>>> >> def get_instance_dict(self, class_, instance):
>>> >> return instance.__dict__
>>> >>
>>> >> def initialize_instance_dict(self, class_, instance):
>>> >> instance.reset_traits()
>>> >>
>>> >> def initialize_collection(self, key, state, factory):
>>> >> data = factory()
>>> >> return ObjectCollectionAdapter(key, state, data), data
>>> >>
>>> >> def install_state(self, class_, instance, state):
>>> >> self.states[id(instance)] = state
>>> >>
>>> >> def state_getter(self, class_):
>>> >> def find(instance):
>>> >> return self.states[id(instance)]
>>> >> return find
>>> >>
>>> >>
>>> >> class ObjectCollectionAdapter(object):
>>> >> """
>>> >> An adapter for SQLAlchemy for TraitsListObject which is
>>> >> the collection
>>> >> we use for storing instances of classes within
>>> >> attributes of other
>>> >> classes.
>>> >>
>>> >> TODO: Think of a way to get this to behave like a
>>> >> normal traited
>>> >> attribute of objects.
>>> >> At the moment collection attributes are set through the
>>> >> mapper by SA
>>> >> when using relations or backref.
>>> >>
>>> >> """
>>> >> def __init__(self, key, state, collection):
>>> >> self.key = key
>>> >> self.state = state
>>> >> self.collection = collection
>>> >> setattr(collection, '_sa_adapter', self)
>>> >>
>>> >> def unlink(self, data):
>>> >> setattr(data, '_sa_adapter', None)
>>> >>
>>> >> def adapt_like_to_iterable(self, obj):
>>> >> return iter(obj)
>>> >>
>>> >> def append_with_event(self, item, initiator=None):
>>> >> self.collection.append(item, emit=initiator)
>>> >>
>>> >> def append_without_event(self, item):
>>> >> self.collection.append(item, emit=False)
>>> >>
>>> >> def remove_with_event(self, item, initiator=None):
>>> >> self.collection.remove(item, emit=initiator)
>>> >>
>>> >> def remove_without_event(self, item):
>>> >> self.collection.remove(item, emit=False)
>>> >>
>>> >> def clear_with_event(self, initiator=None):
>>> >> for item in list(self):
>>> >> self.remove_with_event(item, initiator)
>>> >>
>>> >> def clear_without_event(self):
>>> >> for item in list(self):
>>> >> self.remove_without_event(item)
>>> >>
>>> >> def __iter__(self):
>>> >> return iter(self.collection)
>>> >>
>>> >> def fire_append_event(self, item, initiator=None):
>>> >> if initiator is not False and item is not None:
>>> >> self.state.get_impl(self.key).fire_append_event
>>> >> (self.state, item,
>>> >>
>>> >> initiator)
>>> >>
>>> >> def fire_remove_event(self, item, initiator=None):
>>> >> if initiator is not False and item is not None:
>>> >> self.state.get_impl(self.key).fire_remove_event
>>> >> (self.state, item,
>>> >>
>>> >> initiator)
>>> >>
>>> >> def fire_pre_remove_event(self, initiator=None):
>>> >> self.state.get_impl(self.key).fire_pre_remove_event
>>> >> (self.state,
>>> >>
>>> >> initiator)
>>> >>
>>> >> class ObjectCollection(TraitListObject):
>>> >> __emulates__ = list
>>> >>
>>> >> def __init__( self, trait, object, name, value ):
>>> >> pass
>>> >>
>>> >> def __init__(self):
>>> >> self.members = list()
>>> >> def append(self, object, emit=None):
>>> >> self.members.append(object)
>>> >> #collection_adapter(self).fire_append_event(object,
>>> >> emit) def remove(self, object, emit=None):
>>> >> #collection_adapter(self).fire_pre_remove_event(object)
>>> >> self.members.remove(object)
>>> >> #collection_adapter(self).fire_remove_event(object,
>>> >> emit) def __getitem__(self, index):
>>> >> return self.members[index]
>>> >> def __iter__(self):
>>> >> return iter(self.members)
>>> >> def __len__(self):
>>> >> return len(self.members)
>>> >> def __repr__(self):
>>> >> if len(self.members) == 0:
>>> >> return '[empty collection]'
>>> >> s = '['
>>> >> for item in self.members[0:max(5, len(self.members))]:
>>> >> s += '%s,' % item
>>> >> s += '...]'
>>> >> return s
>>> >>
>>> >>
>>> >> class DeclarativeMetaTraits(MetaHasTraits):
>>> >> pass
>>> >>
>>> >> # def __new__(cls, classname, bases, dict_):
>>> >> # return MetaHasTraits.__new__(cls, classname, bases,
>>> >> dict_)
>>> >>
>>> >> def __init__(cls, classname, bases, dict_):
>>> >>
>>> >> if '_decl_class_registry' in cls.__dict__:
>>> >> super(DeclarativeMetaTraits, cls).__init__(cls,
>>> >> classname, bases, dict_)
>>> >> return
>>> >>
>>> >> if '__tablename__' not in dict_:
>>> >> setattr(cls, '__tablename__', cls.__name__)
>>> >>
>>> >> # create sql columns from flagged traits
>>> >> if '__class_traits__' in cls.__dict__:
>>> >> traits = cls.__dict__['__class_traits__']
>>> >> for key, trait in traits.iteritems():
>>> >> if getattr( trait, 'sqldb' ):
>>> >> args = getattr( trait, 'col_args' ) or ()
>>> >> kwargs = getattr( trait, 'col_kwargs' ) or
>>> >> {} if 'name' not in kwargs:
>>> >> kwargs['name'] = key
>>> >> if 'type_' not in kwargs:
>>> >> kwargs['type_'] =
>>> >> eval(TRAIT_MAPPING[type (trait.trait_type)])
>>> >>
>>> >> if getattr(trait, 'sqlpk'):
>>> >> kwargs['primary_key'] = True
>>> >>
>>> >> c = Column(*args, **kwargs)
>>> >> dict_[key] = c
>>> >> setattr(cls, key, c)
>>> >>
>>> >> # Pass this along to SA's declarative mapper stuff
>>> >> _as_declarative(cls, classname, dict_)
>>> >>
>>> >> super(DeclarativeMetaTraits, cls).__init__(cls,
>>> >> classname, bases, dict_)
>>> >>
>>> >>
>>> >> class HasTraitsORM(HasTraits):
>>> >> __metaclass__ = DeclarativeMetaTraits
>>> >>
>>> >> # some SA hacking
>>> >> __sa_instrumentation_manager__ = HasTraitsORMState
>>> >>
>>> >> _decl_class_registry = dict()
>>> >> metadata = MetaData()
>>> >>
>>> >> # Any implicit traits added by SQLAlchemy are transient and
>>> >> should not be
>>> >> # copied through .clone_traits(), copy.copy(), or pickling.
>>> >> _ = Python(transient=True)
>>> >>
>>> >>
>>> >> # For some reason we need to override these, our superclass
>>> >> screws things up
>>> >>
>>> >> def __getattr__(self, key):
>>> >> if is_instrumented(self, key):
>>> >> return get_attribute(self, key)
>>> >> else:
>>> >> try:
>>> >> return getattr(self, key)
>>> >> except:
>>> >> pass
>>> >>
>>> >> def __setattr__(self, key, value):
>>> >> if is_instrumented(self, key):
>>> >> set_attribute(self, key, value)
>>> >> else:
>>> >> setattr(self, key, value)
>>> >>
>>> >>
>>> >> #################################################
>>>
>>>
>>
>>
>> >>
>>
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To post to this group, send email to sqlalchemy@googlegroups.com
To unsubscribe from this group, send email to
sqlalchemy+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en
-~----------~----~----~----~------~----~------~--~---
