[sqlalchemy] Re: Migrating objects across sessions
On Jan 17, 2009, at 1:46 PM, Darren Govoni wrote: > > Hi, > Thanks for the suggestions. In my system, it is a federation of > databases using SQLA as the "object layer". There are message queues > that move detched data around as needed. My goal is to keep the > application as OO as possible with respect to interacting with the > database, queuing etc. In these cases, objects are retrieved from some > databases and need to be saved to others. > > there's a real benefit to not mixing and matching strategies since its > real nice how SQLA already mitigates the complexities of SQL under > complex objects. > > That said, could it be a humble feature request down the road to have > implicit __deepcopy__ implemented in the instrumented SQLA code > injected > into objects such that it provides this capability. and something > intuitive to exercise it like. > > copiedobjectinnewsession = session.migrate(currentobject) > > This could require some interface support on the objects if need be. > > Since SQLA is already capable of managing the object graph and state > it > would seem logical to re-use those mechanics to implement this? > > Or am I one foot off my rocker? It's a use case that comes up now and then, so I'd implement it as a SQLA extension and it would perhaps call into the attributes API directly to set the appropriate instance state for the new copies. It would likely be a mixin that implements __deepcopy__(). --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Migrating objects across sessions
Hi,
Thanks for the suggestions. In my system, it is a federation of
databases using SQLA as the "object layer". There are message queues
that move detched data around as needed. My goal is to keep the
application as OO as possible with respect to interacting with the
database, queuing etc. In these cases, objects are retrieved from some
databases and need to be saved to others.
there's a real benefit to not mixing and matching strategies since its
real nice how SQLA already mitigates the complexities of SQL under
complex objects.
That said, could it be a humble feature request down the road to have
implicit __deepcopy__ implemented in the instrumented SQLA code injected
into objects such that it provides this capability. and something
intuitive to exercise it like.
copiedobjectinnewsession = session.migrate(currentobject)
This could require some interface support on the objects if need be.
Since SQLA is already capable of managing the object graph and state it
would seem logical to re-use those mechanics to implement this?
Or am I one foot off my rocker?
Cheers!
On Sat, 2009-01-17 at 12:00 -0500, Michael Bayer wrote:
> I advocate using an explicit copy method for this use case. Its true
> that you can't just copy the full __dict__ of the object along.
>
> At its core the state which is required is that the object has an
> "_sa_instance_state" attribute which represents a transient object (as
> opposed to a persistent state, which is what you get when you load the
> object). If you copy the existing _sa_instance_state, SQLA will see
> the object as already persistent in the database.
>
> The easiest way to get that state is to just instantiate a new object
> normally, i.e. newobject = MyObject(). You then want to get the
> correct attribute state present, while at the same time triggering the
> "change" events which SQLAlchemy uses to determine what data needs to
> be inserted. For that you set an attribute, i.e. newobject.foo =
> 'bar' or setattr(newobject, 'foo', 'bar').
>
> Hence doing it explcitly, like by using __deepcopy__ (see the python
> docs for copy.deepcopy), makes all this as straightforward as possible:
>
> class MyClass(object):
> def __deepcopy__(self, memo):
> newobj = MyClass()
>
> # copy scalar attributes
> for attr in ('foo', 'bar', 'bat'):
> setattr(newobj, attr, getattr(self, attr))
>
> # copy references to other objects
> for ref in ('x', 'y', 'z'):
> setattr(newobj, attr, deepcopy(getattr(self, attr)))
>
> return newobj
>
>
> If you really want to copy lots of data between two databases though,
> its probably a lot more efficient to not use an ORM for that, and to
> use either explicit INSERT constructs or the bulk update tools
> supplied with the database.
>
>
> On Jan 17, 2009, at 8:06 AM, Darren Govoni wrote:
>
> >
> > Yes, i'd like to do that, but because of the code injection seems to
> > carry state from the old session, the new session has trouble with
> > even
> > the copy.
> >
> > On Sat, 2009-01-17 at 15:23 +0200, a...@svilendobrev.com wrote:
> >> read various threads about copy/deep-copy.
> >> in any case u'll have to do the hierarchy-copying yourself - even if
> >> moving one object (alone) from session to session succeeds somehow.
> >>
> >> On Saturday 17 January 2009 14:47:41 Darren Govoni wrote:
> >>> After some further experimenting with this, it seems I cannot take
> >>> a mapped object I retrieved from one session and save it in
> >>> another.
> >>>
> >>> Using expunge then adding the object to a second session on a
> >>> different database does not work. I thought maybe it would.
> >>>
> >>> This is a really useful thing to do, especially for replication at
> >>> the object layer.
> >>>
> >>> So is it true that this cannot currently be done with sqlalchemy? I
> >>> have been reading the docs, but no clues yet.
> >>>
> >>> On Fri, 2009-01-16 at 16:53 -0500, Darren Govoni wrote:
> Hi,
> I have an object (with references) I get from a session on
> database A and I want to copy it (deep copy) to database B. I
> tried expunging it from the first session and adding it to the
> second. What's the best practice to do this?
>
> thanks!
> Darren
> >>
> >>>
> >
> >
> > >
>
>
> >
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To post to this group, send email to sqlalchemy@googlegroups.com
To unsubscribe from this group, send email to
sqlalchemy+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en
-~--~~~~--~~--~--~---
[sqlalchemy] Re: Migrating objects across sessions
I advocate using an explicit copy method for this use case. Its true
that you can't just copy the full __dict__ of the object along.
At its core the state which is required is that the object has an
"_sa_instance_state" attribute which represents a transient object (as
opposed to a persistent state, which is what you get when you load the
object). If you copy the existing _sa_instance_state, SQLA will see
the object as already persistent in the database.
The easiest way to get that state is to just instantiate a new object
normally, i.e. newobject = MyObject(). You then want to get the
correct attribute state present, while at the same time triggering the
"change" events which SQLAlchemy uses to determine what data needs to
be inserted. For that you set an attribute, i.e. newobject.foo =
'bar' or setattr(newobject, 'foo', 'bar').
Hence doing it explcitly, like by using __deepcopy__ (see the python
docs for copy.deepcopy), makes all this as straightforward as possible:
class MyClass(object):
def __deepcopy__(self, memo):
newobj = MyClass()
# copy scalar attributes
for attr in ('foo', 'bar', 'bat'):
setattr(newobj, attr, getattr(self, attr))
# copy references to other objects
for ref in ('x', 'y', 'z'):
setattr(newobj, attr, deepcopy(getattr(self, attr)))
return newobj
If you really want to copy lots of data between two databases though,
its probably a lot more efficient to not use an ORM for that, and to
use either explicit INSERT constructs or the bulk update tools
supplied with the database.
On Jan 17, 2009, at 8:06 AM, Darren Govoni wrote:
>
> Yes, i'd like to do that, but because of the code injection seems to
> carry state from the old session, the new session has trouble with
> even
> the copy.
>
> On Sat, 2009-01-17 at 15:23 +0200, a...@svilendobrev.com wrote:
>> read various threads about copy/deep-copy.
>> in any case u'll have to do the hierarchy-copying yourself - even if
>> moving one object (alone) from session to session succeeds somehow.
>>
>> On Saturday 17 January 2009 14:47:41 Darren Govoni wrote:
>>> After some further experimenting with this, it seems I cannot take
>>> a mapped object I retrieved from one session and save it in
>>> another.
>>>
>>> Using expunge then adding the object to a second session on a
>>> different database does not work. I thought maybe it would.
>>>
>>> This is a really useful thing to do, especially for replication at
>>> the object layer.
>>>
>>> So is it true that this cannot currently be done with sqlalchemy? I
>>> have been reading the docs, but no clues yet.
>>>
>>> On Fri, 2009-01-16 at 16:53 -0500, Darren Govoni wrote:
Hi,
I have an object (with references) I get from a session on
database A and I want to copy it (deep copy) to database B. I
tried expunging it from the first session and adding it to the
second. What's the best practice to do this?
thanks!
Darren
>>
>>>
>
>
> >
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To post to this group, send email to sqlalchemy@googlegroups.com
To unsubscribe from this group, send email to
sqlalchemy+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en
-~--~~~~--~~--~--~---
[sqlalchemy] Re: Migrating objects across sessions
> Yes, i'd like to do that, but because of the code injection seems > to carry state from the old session, the new session has trouble > with even the copy. then, separate the issues: a) make a working deep copy within same session (this is not at all trivial, except for very simple schemas) b) make a working move-across-session for singular obj and just then combine > On Sat, 2009-01-17 at 15:23 +0200, a...@svilendobrev.com wrote: > > read various threads about copy/deep-copy. > > in any case u'll have to do the hierarchy-copying yourself - even > > if moving one object (alone) from session to session succeeds > > somehow. > > > > On Saturday 17 January 2009 14:47:41 Darren Govoni wrote: > > > After some further experimenting with this, it seems I cannot > > > take a mapped object I retrieved from one session and save it > > > in another. > > > > > > Using expunge then adding the object to a second session on a > > > different database does not work. I thought maybe it would. > > > > > > This is a really useful thing to do, especially for replication > > > at the object layer. > > > > > > So is it true that this cannot currently be done with > > > sqlalchemy? I have been reading the docs, but no clues yet. > > > > > > On Fri, 2009-01-16 at 16:53 -0500, Darren Govoni wrote: > > > > Hi, > > > > I have an object (with references) I get from a session on > > > > database A and I want to copy it (deep copy) to database B. I > > > > tried expunging it from the first session and adding it to > > > > the second. What's the best practice to do this? > > > > > > > > thanks! > > > > Darren > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Migrating objects across sessions
I tried using session.merge(deepcopy) on my copy.deepcopy of a mapped object(s) and when I commit the session I get RuntimeError: maximum recursion depth exceeded strange. On Sat, 2009-01-17 at 15:23 +0200, a...@svilendobrev.com wrote: > read various threads about copy/deep-copy. > in any case u'll have to do the hierarchy-copying yourself - even if > moving one object (alone) from session to session succeeds somehow. > > On Saturday 17 January 2009 14:47:41 Darren Govoni wrote: > > After some further experimenting with this, it seems I cannot take > > a mapped object I retrieved from one session and save it in > > another. > > > > Using expunge then adding the object to a second session on a > > different database does not work. I thought maybe it would. > > > > This is a really useful thing to do, especially for replication at > > the object layer. > > > > So is it true that this cannot currently be done with sqlalchemy? I > > have been reading the docs, but no clues yet. > > > > On Fri, 2009-01-16 at 16:53 -0500, Darren Govoni wrote: > > > Hi, > > > I have an object (with references) I get from a session on > > > database A and I want to copy it (deep copy) to database B. I > > > tried expunging it from the first session and adding it to the > > > second. What's the best practice to do this? > > > > > > thanks! > > > Darren > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Migrating objects across sessions
Yes, i'd like to do that, but because of the code injection seems to carry state from the old session, the new session has trouble with even the copy. On Sat, 2009-01-17 at 15:23 +0200, a...@svilendobrev.com wrote: > read various threads about copy/deep-copy. > in any case u'll have to do the hierarchy-copying yourself - even if > moving one object (alone) from session to session succeeds somehow. > > On Saturday 17 January 2009 14:47:41 Darren Govoni wrote: > > After some further experimenting with this, it seems I cannot take > > a mapped object I retrieved from one session and save it in > > another. > > > > Using expunge then adding the object to a second session on a > > different database does not work. I thought maybe it would. > > > > This is a really useful thing to do, especially for replication at > > the object layer. > > > > So is it true that this cannot currently be done with sqlalchemy? I > > have been reading the docs, but no clues yet. > > > > On Fri, 2009-01-16 at 16:53 -0500, Darren Govoni wrote: > > > Hi, > > > I have an object (with references) I get from a session on > > > database A and I want to copy it (deep copy) to database B. I > > > tried expunging it from the first session and adding it to the > > > second. What's the best practice to do this? > > > > > > thanks! > > > Darren > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Migrating objects across sessions
read various threads about copy/deep-copy. in any case u'll have to do the hierarchy-copying yourself - even if moving one object (alone) from session to session succeeds somehow. On Saturday 17 January 2009 14:47:41 Darren Govoni wrote: > After some further experimenting with this, it seems I cannot take > a mapped object I retrieved from one session and save it in > another. > > Using expunge then adding the object to a second session on a > different database does not work. I thought maybe it would. > > This is a really useful thing to do, especially for replication at > the object layer. > > So is it true that this cannot currently be done with sqlalchemy? I > have been reading the docs, but no clues yet. > > On Fri, 2009-01-16 at 16:53 -0500, Darren Govoni wrote: > > Hi, > > I have an object (with references) I get from a session on > > database A and I want to copy it (deep copy) to database B. I > > tried expunging it from the first session and adding it to the > > second. What's the best practice to do this? > > > > thanks! > > Darren --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Migrating objects across sessions
After some further experimenting with this, it seems I cannot take a mapped object I retrieved from one session and save it in another. Using expunge then adding the object to a second session on a different database does not work. I thought maybe it would. This is a really useful thing to do, especially for replication at the object layer. So is it true that this cannot currently be done with sqlalchemy? I have been reading the docs, but no clues yet. On Fri, 2009-01-16 at 16:53 -0500, Darren Govoni wrote: > Hi, > I have an object (with references) I get from a session on database A > and I want to copy it (deep copy) to database B. I tried expunging it > from the first session and adding it to the second. What's the best > practice to do this? > > thanks! > Darren > > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
