Re: [Zope-dev] Experiments with ORMapping

2001-05-15 Thread Shane Hathaway

On Mon, 14 May 2001, ender wrote:

> i want to thank DC (jim, shane, and paul) for inviting me to come to the new
> DC offices. i had a great time and learned a bunch... and met
> the BFDL.

It was good to talk with you!

> i gave a quick overview of the smartobjects design/framework and  jim and
> shane presented some ideas for an or mapping for the zodb.
>
> to be honest it seemed both extremely elegant and slightly disturbing at the
> sametime. I've grown accustomed to treating the ZODB as a magic black box,
> that just works. integrating an or layer at this low level disturbs me as it
> meant that application developers would need to tinker at the lowest level of
> the zope framework, plus the dangers that to properly support application
> development the zodb would need to be bloated with accessory apis. at the
> same time it offers a really elegant way to make the or mapping available to
> python programmers on a pure python level (which is something i'm also
> interested in).

Jim's ideas are often disturbingly elegant and difficult to understand all
at the same time. :-)

This reminds me of a recent slashdot thread that talked about a
photographer in the days of black and white film who had devised a scheme
for preserving color, and the article presented the results.  It was
disturbing for readers to realize that life in past centuries actually was
in color!

Programmers really wouldn't have to tinker with this layer.  They would
just write a schema and a little glue code that this layer understands.

> for both approaches there are some similiar problems that must be tackled,
> namely definition of the or mapping, generation of stub python classes,
> integration with zope notions (permission, etc). so hopefully discussion here
> will flesh out, to the point of making a choice clear.
>
> i'm primarily focused on doing the or mapping definition in xml, as its
> language neutral and allows me to leverage other tools (like torque
> java.apache.org/turbine/torque.html) to assist in application
> development/porting.
>
> i've thought about it for a few days, and while i'm not convinced completely
> that a zodb approach is the best method, i'm going to start work on the some
> of the support classes nesc. for moving forward, namely some of  the xml
> model/generators outlined in my design (embryonic)
> (http://www.zope.org/Members/k_vertigo/smartobjects)
>
> most of my concerns echo the issues that philip has brought up re the
> need of application developers. i do feel some if not most of these issues
> can be solved, i'm just wondering if for zope this is the best solution. for
> pure python development it seems alright. for zope, i wonder if a higher
> level application framework is more suitable.

It all depends on what layer you're developing for, I guess.  In the web
interface you can't use rich object hierarchies as fluidly as you can in
Python.  On the web level you don't care as much about transparent
persistence, but in Python you do.

> to clarify some existing points in this thread to date.
>
> - (IMO) zpatterns is about two things, delegation and data abstraction at
> the application level. the zpatterns framework doesn't assist in automating
> this and its up to the integrator/developer to manually wire together all the
> app/business objects. the smartobjects framework is about automating
> development and presenting easier development interfaces for both
> developers and designers. this isn't to say that such automation and
> interfaces couldn't be built on top of zpatterns, just to say that there are
> some different design goals.

Let's say you want CMF objects to be stored in an RDBMS.  You don't want
to use ZODB stubs and you want a tree of persistent objects.  ZPatterns
can't help you.  Can SmartObjects?

> - one of the primary goals of smartobjects is easy integration with zope in a
> manner that is natural for zope developers. this does not mandate a new db
> access api, its about facilitating automation of this api so that the objects
> can be treated as normaly zope objects. both approaches (smartobjects and or
> zodb) must still tackle similiar fundamental issues to integrate with zope.
> on the zodb level this seems complicated to me (which betrays my incomplete
> understanding of the zodb internals), plus it requires some additions to the
> zodb interface to allow effective application usage (namely a generic query
> mechanism, and probably interogation of a class information).

Here's another way to look at this: if SmartObjects doesn't use code from
the ZODB, it could be reinventing parts of it.  It seems like a good idea
to get to know ZODB well before proceeding too far.

> - one of my main objectives in developing an or mapping is to use the
> acs4 (www.arsdigita.com) applications natively (or as close as possible)
> within zope.

Sounds great!

> - my smartobjects design (www.zope.org/Members/k_vertigo/smartobjects) is
> mine, in that smartobjects is primarily a iuveno 

Re: [Zope-dev] Experiments with ORMapping

2001-05-14 Thread ender

Hello,

i want to thank DC (jim, shane, and paul) for inviting me to come to the new 
DC offices. i had a great time and learned a bunch... and met 
the BFDL.

i gave a quick overview of the smartobjects design/framework and  jim and 
shane presented some ideas for an or mapping for the zodb.

to be honest it seemed both extremely elegant and slightly disturbing at the 
sametime. I've grown accustomed to treating the ZODB as a magic black box, 
that just works. integrating an or layer at this low level disturbs me as it 
meant that application developers would need to tinker at the lowest level of 
the zope framework, plus the dangers that to properly support application 
development the zodb would need to be bloated with accessory apis. at the 
same time it offers a really elegant way to make the or mapping available to 
python programmers on a pure python level (which is something i'm also 
interested in). 

for both approaches there are some similiar problems that must be tackled, 
namely definition of the or mapping, generation of stub python classes, 
integration with zope notions (permission, etc). so hopefully discussion here 
will flesh out, to the point of making a choice clear.

i'm primarily focused on doing the or mapping definition in xml, as its 
language neutral and allows me to leverage other tools (like torque 
java.apache.org/turbine/torque.html) to assist in application 
development/porting.

i've thought about it for a few days, and while i'm not convinced completely
that a zodb approach is the best method, i'm going to start work on the some 
of the support classes nesc. for moving forward, namely some of  the xml 
model/generators outlined in my design (embryonic)
(http://www.zope.org/Members/k_vertigo/smartobjects)

most of my concerns echo the issues that philip has brought up re the 
need of application developers. i do feel some if not most of these issues
can be solved, i'm just wondering if for zope this is the best solution. for
pure python development it seems alright. for zope, i wonder if a higher 
level application framework is more suitable. 

to clarify some existing points in this thread to date.

- (IMO) zpatterns is about two things, delegation and data abstraction at
the application level. the zpatterns framework doesn't assist in automating
this and its up to the integrator/developer to manually wire together all the 
app/business objects. the smartobjects framework is about automating
development and presenting easier development interfaces for both 
developers and designers. this isn't to say that such automation and 
interfaces couldn't be built on top of zpatterns, just to say that there are 
some different design goals. 

- one of the primary goals of smartobjects is easy integration with zope in a
manner that is natural for zope developers. this does not mandate a new db 
access api, its about facilitating automation of this api so that the objects 
can be treated as normaly zope objects. both approaches (smartobjects and or 
zodb) must still tackle similiar fundamental issues to integrate with zope. 
on the zodb level this seems complicated to me (which betrays my incomplete 
understanding of the zodb internals), plus it requires some additions to the 
zodb interface to allow effective application usage (namely a generic query 
mechanism, and probably interogation of a class information). 

- one of my main objectives in developing an or mapping is to use the
acs4 (www.arsdigita.com) applications natively (or as close as possible) 
within zope.

- my smartobjects design (www.zope.org/Members/k_vertigo/smartobjects) is
mine, in that smartobjects is primarily a iuveno project this does not mean 
it is smartobjects. stephen richter (of iuveno) liked parts of the design and 
will likely use it parts of it in a iuveno implementation but what comes out 
of iuveno is not limited or beholden in anyway by my proposal. the key focus 
of the proposal is a metamodel (schema) based approach.

questions

- i think any integration of or layer should be complementary to the zope 
framework. shane's existing examples and discussion of an or layer seem to be 
more geared towards replacing the zodb, than complementing. is the idea to 
use some sort of zodb product as a mounted storage?, or is this a replacement 
strategy.

cheers

kapil

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Experiments with ORMapping

2001-05-14 Thread Joachim Werner

> My thought on this, is that you will be forced to explicitly consider the
> nature of this relationship and its storage at the application level.  If
> you write explicitly for ZODB, you might use a BTree, for example.  Or
> perhaps you'd have some kind of global container for the Tasks, with a
> Catalog to index them, and a method on TaskPerformer to query the Catalog.
>
> How would you propose to map this to an RDMS?  Well, substituting catalog
> queries might be relatively straightforward, but perhaps rather time
> consuming to develop in a generic way.  The BTree might give you a bit
more
> trouble.  Probably you'd have to end up using some sort of generic
> queryable containers that translated to BTrees or Catalogs or RDBMS
tables,
> depending...
>
> And lo!  You now have an application framework.  Oops.

I'll try to somehow get your and Shane's opinions together a bit: I think
Shane's approach does most of the job (of OR mapping) if the objects we are
talking about are relatively simple and the application domain is relatively
typical for what Zope can do well. E.g. the composite objects we have in
mind for the groupware (user credentials coming from LDAP, rest in the ZODB,
or file stored in the filesystem metadata in SQL DB, rest in ZODB) can
easily be modelled this way. And the existing Zope API (manage_AddX,
manage_changeProperties etc.) would do most of the job.

But the problems start with querying. Can the ZCatalog do it? Does it make
sense to implement a "SQLZCatalog" instead of just offering native SQL query
objects or an abstract query language like SkinSkript? To put it simply: The
reason why we want to develop a "framework" (or framework extension) is that
the Zope framework just doesn't do the stuff we need yet. BTW, the reason
why we didn't just use ZPatterns was very similar: ZPatterns makes it easy
to write an abstract implementation of a project, but you still have to
write all the "glue code" between the storage and the application layer by
hand. What we want (and what TransWarp is about, with a slightly different
focus) is that we can just map an attribute to a data source and all the
glue code (either ZODB or SQL statements) is auto-generated.

If the new "framework" just extends the existing Zope API, I don't see a
problem. ZPatterns did not, but maybe it is just not that easy to do so (and
not all parts of the Zope API really should bear this name ...).

The really good thing about the whole discussion is that at the end we will
see much better where we should put things. We (the SmartObjects people at
iuveno) do not really want to have a completely new world of SmartObjects
that will require a new API to learn. We just want to add new functionality
to Zope, be it in the core or as mix-in classes, or through-the-web
products.

Joachim


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Experiments with ORMapping

2001-05-14 Thread Joachim Werner

> My thought on this, is that you will be forced to explicitly consider the
> nature of this relationship and its storage at the application level.  If
> you write explicitly for ZODB, you might use a BTree, for example.  Or
> perhaps you'd have some kind of global container for the Tasks, with a
> Catalog to index them, and a method on TaskPerformer to query the Catalog.
>
> How would you propose to map this to an RDMS?  Well, substituting catalog
> queries might be relatively straightforward, but perhaps rather time
> consuming to develop in a generic way.  The BTree might give you a bit
more
> trouble.  Probably you'd have to end up using some sort of generic
> queryable containers that translated to BTrees or Catalogs or RDBMS
tables,
> depending...
>
> And lo!  You now have an application framework.  Oops.

I'll try to somehow get your and Shane's opinions together a bit: I think
Shane's approach does most of the job (of OR mapping) if the objects we are
talking about are relatively simple and the application domain is relatively
typical for what Zope can do well. E.g. the composite objects we have in
mind for the groupware (user credentials coming from LDAP, rest in the ZODB,
or file stored in the filesystem metadata in SQL DB, rest in ZODB) can
easily be modelled this way. And the existing Zope API (manage_AddX,
manage_changeProperties etc.) would do most of the job.

But the problems start with querying. Can the ZCatalog do it? Does it make
sense to implement a "SQLZCatalog" instead of just offering native SQL query
objects or an abstract query language like SkinSkript? To put it simply: The
reason why we want to develop a "framework" (or framework extension) is that
the Zope framework just doesn't do the stuff we need yet. BTW, the reason
why we didn't just use ZPatterns was very similar: ZPatterns makes it easy
to write an abstract implementation of a project, but you still have to
write all the "glue code" between the storage and the application layer by
hand. What we want (and what TransWarp is about, with a slightly different
focus) is that we can just map an attribute to a data source and all the
glue code (either ZODB or SQL statements) is auto-generated.

If the new "framework" just extends the existing Zope API, I don't see a
problem. ZPatterns did not, but maybe it is just not that easy to do so (and
not all parts of the Zope API really should bear this name ...).

The really good thing about the whole discussion is that at the end we will
see much better where we should put things. We (the SmartObjects people at
iuveno) do not really want to have a completely new world of SmartObjects
that will require a new API to learn. We just want to add new functionality
to Zope, be it in the core or as mix-in classes, or through-the-web
products.

Joachim


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Experiments with ORMapping

2001-05-14 Thread Joachim Werner

> Now, it may be useful to provide a management interface for defining the
> schema mapping.  I haven't approached that yet; AFAICT this is where the
> work done on SmartObjects and DBObjects would be very useful.  Initially
> I was planning for people to code the mapping purely in Python so we
> could gain experience and find common patterns before inventing a UI.

Our initial plans for SmartObjects (if I get Stephan right) are to define
all objects in Python code (as DBObjects do now), but to also provide two
alternative ways of creating the necessary code:

a) by importing and "compiling" an XML object description (which might come
from a UML tool)
b) via a ZMI interface similar to the ZClass GUI (which may or may not use
the XML schemas, depending on the actual implementation)

Of course it would be necessary to dynamically refresh the newly generated
filesystem-based code. The other issue that applies in general to all
filesystem-based code is that it is not directly available over ZEO as it is
not in the ZODB. But that should not be a very large problem ...


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Experiments with ORMapping

2001-05-14 Thread Shane Hathaway

"Phillip J. Eby" wrote:
> All this is of course just my opinion, and quite possibly wrong.  But I
> think that an application developer would get more mileage out of your
> DBAPI product (coupled with some utility classes to replace Racks) or from
> ZPatterns than they would from this approach to O-R mapping.

Your opinion is highly valued, and I agree there are unknown quantities
at this point.  But I think the unknowns are not significant enough to
drop the idea.  The goal is transparent persistence with an arbitrary
RDBMS schema.  Many developers prefer to code for ZODB but customers
demand SQL.  This would bridge the gap.

Regarding performance, this method is actually ideal IMHO.  Data is read
once, converted to an object, and kept for later connections, just like
ZODB.

Also, Jim has suggested database-specific features for querying, etc.
and abstracting these does require application-level logic.  The goal is
not to avoid all application logic, just to move as much as possible to
the storage layer.  (My statement about this approach not using any
application logic was thus inaccurate.)

Shane

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Experiments with ORMapping

2001-05-14 Thread Phillip J. Eby

At 12:26 PM 5/14/01 -0400, Shane Hathaway wrote:
>Chris Withers wrote:
> >
> > Shane Hathaway wrote:
> > >
> > > One would define an "ObjectMappingSchema" whose job it is to store and
> > > retrieve objects of a specific type and in a specific location.  It
> > > would usually grab a database connection object to do its work.  When
> > > loading, it would perform a query then manually put attributes into a
> > > persistent object.  When storing, it would grab specific attributes from
> > > the persistent object and execute a statement to store those attributes.
> >
> > How does this differ from ZPatterns? Phil?
>
>ZPatterns implements storage logic on the application level.
>Applications have to be aware of (in fact they have to be centered
>around) ZPatterns.  This alternate approach keeps storage logic
>independent of application logic.  It lets you take any code written for
>the ZODB and move it to your RDBMS schema without changing the code
>(most of the time :-) ).

I am reminded of a passage from "Dirk Gently's Holistic Detective Agency", 
wherein Dirk decides to cut to the heart of the problem, and simply writes 
down the answer.  Now, he declares, he is faced with only a relatively much 
easier problem: What strange language did he write the answer down in?

While I'm not saying that what you're doing isn't possible, I *will* say 
that I believe you are replacing a small amount of API-specificity in the 
code with a similar amount of specificity, coupled with a much larger 
complexity in the mapping layers.  It is, IMHO, much harder to write 
generic mappings at the schema layer where you propose, than to do so at 
the domain logic layer where ZPatterns operates.  Ty and I spent many weeks 
chewing over designs in the space you're talking about, back before 
ZPatterns existed, after consulting at length with Jim Fulton and Michel 
Pelletier.  Our conclusion was that yes, it was *possible* to do the 
mapping you're talking about, but that it was very high impedance for the 
kinds of applications we had in mind.

I'll give a simple example to show what I'm talking about.  Consider a task 
framework where TaskPerformers are assigned Tasks, in a simple one-to-many 
relationship.  TaskPerformers, in this framework, perform up to hundreds of 
tasks per day, thousands per month.  Assuming an RDBMS backend, how do you 
propose to map this in your storage model?

My thought on this, is that you will be forced to explicitly consider the 
nature of this relationship and its storage at the application level.  If 
you write explicitly for ZODB, you might use a BTree, for example.  Or 
perhaps you'd have some kind of global container for the Tasks, with a 
Catalog to index them, and a method on TaskPerformer to query the Catalog.

How would you propose to map this to an RDMS?  Well, substituting catalog 
queries might be relatively straightforward, but perhaps rather time 
consuming to develop in a generic way.  The BTree might give you a bit more 
trouble.  Probably you'd have to end up using some sort of generic 
queryable containers that translated to BTrees or Catalogs or RDBMS tables, 
depending...

And lo!  You now have an application framework.  Oops.

We haven't even addressed performance issues yet, which are the real 
killer.  The only way (again IMHO) to get reasonably high performance that 
is "portable" across different databases is to be able to write code that 
is optimized based on the back-end.  The only way you can push this code 
"under the hood" is to create an all-purpose query mechanism.  Otherwise, 
you must expose enough of the inner workings to the application developer 
that they can include "platform-specific" code for each circumstance.

All this is of course just my opinion, and quite possibly wrong.  But I 
think that an application developer would get more mileage out of your 
DBAPI product (coupled with some utility classes to replace Racks) or from 
ZPatterns than they would from this approach to O-R mapping.


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Experiments with ORMapping

2001-05-14 Thread Shane Hathaway

Chris Withers wrote:
> 
> Shane Hathaway wrote:
> >
> > ZPatterns implements storage logic on the application level.
> > Applications have to be aware of (in fact they have to be centered
> > around) ZPatterns.  This alternate approach keeps storage logic
> > independent of application logic.  It lets you take any code written for
> > the ZODB and move it to your RDBMS schema without changing the code
> > (most of the time :-) ).
> 
> Now that sounds really f^%$^ing cool :-) How can I help?

For now, I can make public the code I wrote for experimentation:

http://www.zope.org/Members/hathawsh/orconn.tar.gz

The archive contains two Python files and a patch.  Put them in the ZODB
directory.  Perform the patch to DB.py.  (It's very minimal.)  Then run
testOR.py.  It just changes a mapping, but it does it transparently and
transactionally. :-)

I need to fishbowl this before going any further.

Shane

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Experiments with ORMapping

2001-05-14 Thread John D. Heintz

A way to approach how this would be useful would be to contrast different use 
environments for ZODB code.

The same code could be used for:
- Unit testing, prototyping, and general exploration could use ZODB with 
DemoStorage and the default Connection objects.

- Production use could make use of ZODB and custom Connection objects that 
map to/from relational tables.

John

On Monday 14 May 2001 09:51, Chris Withers wrote:
> Shane Hathaway wrote:
> > I'm telling you there's a lot more you can do with the code that makes
>
> 
>
> > The next thing to do is to write a fishbowl proposal.
>
> This sounds cool but made my head hurt :-S
>
> Can you try and bring this back down to the level of us mere mortals by
> explaining how your OR stuff would let me take a table of data in an RDBMS
> table and have it appear as objects in the Management Inteferace?
>
> I know that's horribly oversimplified, but some of us only have small
> brains ;-)
>
> cheers,
>
> Chris
>
> ___
> Zope-Dev maillist  -  [EMAIL PROTECTED]
> http://lists.zope.org/mailman/listinfo/zope-dev
> **  No cross posts or HTML encoding!  **
> (Related lists -
>  http://lists.zope.org/mailman/listinfo/zope-announce
>  http://lists.zope.org/mailman/listinfo/zope )

-- 
. . . . . . . . . . . . . . . . . . . . . . . .

John D. Heintz | Senior Engineer

1016 La Posada Dr. | Suite 240 | Austin TX 78752
T 512.633.1198 | [EMAIL PROTECTED]

w w w . d a t a c h a n n e l . c o m

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Experiments with ORMapping

2001-05-14 Thread Chris Withers

Shane Hathaway wrote:
> 
> ZPatterns implements storage logic on the application level.
> Applications have to be aware of (in fact they have to be centered
> around) ZPatterns.  This alternate approach keeps storage logic
> independent of application logic.  It lets you take any code written for
> the ZODB and move it to your RDBMS schema without changing the code
> (most of the time :-) ).

Now that sounds really f^%$^ing cool :-) How can I help?

Chris

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Experiments with ORMapping

2001-05-14 Thread Shane Hathaway

Chris Withers wrote:
> 
> Shane Hathaway wrote:
> >
> > One would define an "ObjectMappingSchema" whose job it is to store and
> > retrieve objects of a specific type and in a specific location.  It
> > would usually grab a database connection object to do its work.  When
> > loading, it would perform a query then manually put attributes into a
> > persistent object.  When storing, it would grab specific attributes from
> > the persistent object and execute a statement to store those attributes.
> 
> How does this differ from ZPatterns? Phil?

ZPatterns implements storage logic on the application level. 
Applications have to be aware of (in fact they have to be centered
around) ZPatterns.  This alternate approach keeps storage logic
independent of application logic.  It lets you take any code written for
the ZODB and move it to your RDBMS schema without changing the code
(most of the time :-) ).

Shane

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Experiments with ORMapping

2001-05-14 Thread John D. Heintz

I think this is a great idea!  I would definetely like to use and contribute 
to this effort.  Having this kind of flexibily would be fantastic.

After demonstratable Python code is working I would request that usability 
issues (UI Schema mapper, data migration/schema evolution tools, ZEO 
integration, multi-Storage uses) be addressed sooner than later.  

John

On Monday 14 May 2001 10:47, Shane Hathaway wrote:
> Chris Withers wrote:
> > Shane Hathaway wrote:
> > > I'm telling you there's a lot more you can do with the code that makes
> >
> > 
> >
> > > The next thing to do is to write a fishbowl proposal.
> >
> > This sounds cool but made my head hurt :-S
> >
> > Can you try and bring this back down to the level of us mere mortals by
> > explaining how your OR stuff would let me take a table of data in an
> > RDBMS table and have it appear as objects in the Management Inteferace?
>
> Sorry, this is at a pretty low level and I do need to explain it better.
>
> One would define an "ObjectMappingSchema" whose job it is to store and
> retrieve objects of a specific type and in a specific location.  It
> would usually grab a database connection object to do its work.  When
> loading, it would perform a query then manually put attributes into a
> persistent object.  When storing, it would grab specific attributes from
> the persistent object and execute a statement to store those attributes.
>
> So let's say you want a ZODB to store and retrieve users in a specific
> table while putting everything else in pickles.  You would create an
> instance of PickleSchema, which implements the ObjectMappingSchema
> interface, and tell it to manage everything *except* the users mapping
> in BasicUserFolder objects.  You would tell it to store and retrieve
> this object using your UserFolderSchema instead.  Your UserFolderSchema
> would store and retrieve the users from the USERS and USER_PREFS
> tables.  The user table wouldn't require the use of OIDs but would
> require unique user IDs.
>
> So in the management interface nothing would change.  Nor would the
> application-level Python code.  You would only define a layer that maps
> objects to a relational database.  You would still see the user folder
> as you do now.
>
> Now, it may be useful to provide a management interface for defining the
> schema mapping.  I haven't approached that yet; AFAICT this is where the
> work done on SmartObjects and DBObjects would be very useful.  Initially
> I was planning for people to code the mapping purely in Python so we
> could gain experience and find common patterns before inventing a UI.
>
> Shane
>
> ___
> Zope-Dev maillist  -  [EMAIL PROTECTED]
> http://lists.zope.org/mailman/listinfo/zope-dev
> **  No cross posts or HTML encoding!  **
> (Related lists -
>  http://lists.zope.org/mailman/listinfo/zope-announce
>  http://lists.zope.org/mailman/listinfo/zope )

-- 
. . . . . . . . . . . . . . . . . . . . . . . .

John D. Heintz | Senior Engineer

1016 La Posada Dr. | Suite 240 | Austin TX 78752
T 512.633.1198 | [EMAIL PROTECTED]

w w w . d a t a c h a n n e l . c o m

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Experiments with ORMapping

2001-05-14 Thread Chris Withers



Shane Hathaway wrote:
> 
> One would define an "ObjectMappingSchema" whose job it is to store and
> retrieve objects of a specific type and in a specific location.  It
> would usually grab a database connection object to do its work.  When
> loading, it would perform a query then manually put attributes into a
> persistent object.  When storing, it would grab specific attributes from
> the persistent object and execute a statement to store those attributes.

How does this differ from ZPatterns? Phil?

> Now, it may be useful to provide a management interface for defining the
> schema mapping.  I haven't approached that yet; AFAICT this is where the
> work done on SmartObjects and DBObjects would be very useful.  Initially
> I was planning for people to code the mapping purely in Python so we
> could gain experience and find common patterns before inventing a UI.

Sounds good to me :-) Thanks for the explanation...

cheers,

Chris

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Experiments with ORMapping

2001-05-14 Thread Shane Hathaway

Chris Withers wrote:
> 
> Shane Hathaway wrote:
> >
> > I'm telling you there's a lot more you can do with the code that makes
> 
> 
> 
> > The next thing to do is to write a fishbowl proposal.
> 
> This sounds cool but made my head hurt :-S
> 
> Can you try and bring this back down to the level of us mere mortals by
> explaining how your OR stuff would let me take a table of data in an RDBMS table
> and have it appear as objects in the Management Inteferace?

Sorry, this is at a pretty low level and I do need to explain it better.

One would define an "ObjectMappingSchema" whose job it is to store and
retrieve objects of a specific type and in a specific location.  It
would usually grab a database connection object to do its work.  When
loading, it would perform a query then manually put attributes into a
persistent object.  When storing, it would grab specific attributes from
the persistent object and execute a statement to store those attributes.

So let's say you want a ZODB to store and retrieve users in a specific
table while putting everything else in pickles.  You would create an
instance of PickleSchema, which implements the ObjectMappingSchema
interface, and tell it to manage everything *except* the users mapping
in BasicUserFolder objects.  You would tell it to store and retrieve
this object using your UserFolderSchema instead.  Your UserFolderSchema
would store and retrieve the users from the USERS and USER_PREFS
tables.  The user table wouldn't require the use of OIDs but would
require unique user IDs.

So in the management interface nothing would change.  Nor would the
application-level Python code.  You would only define a layer that maps
objects to a relational database.  You would still see the user folder
as you do now.

Now, it may be useful to provide a management interface for defining the
schema mapping.  I haven't approached that yet; AFAICT this is where the
work done on SmartObjects and DBObjects would be very useful.  Initially
I was planning for people to code the mapping purely in Python so we
could gain experience and find common patterns before inventing a UI.

Shane

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: oodb philosophics ;) was: Re: [Zope-dev] Experiments with ORMapping

2001-05-14 Thread Juan David Ibáñez Palomar

> 
> > P.S.: Shane, have you developed Symlinks any further? I think they could be
> > extremely useful. I tried out the initial release and liked it, except for
> > the fact that the symlinks looked EXACTLY like real ones, so they can be
> > very irritating ...
> 
> I'm not sure what to do with symlinks.  How should security be applied? 
> How are they most useful?
> 
> It's neat to see it works, though. :-)
> 
> Shane
> 
> 

Last year I needed symlinks and developed a version of your product
where the symlink had a different id than the real object. But the
general solution of a symlink where an arbitrary set of attributes
are from the symlink instance instead of from the real object was
too much difficult for me, so this year we've changed the dessign
of the application and don't need symlinks anymore.


jdavid

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Experiments with ORMapping

2001-05-14 Thread Chris Withers

Shane Hathaway wrote:
> 
> I'm telling you there's a lot more you can do with the code that makes



> The next thing to do is to write a fishbowl proposal.

This sounds cool but made my head hurt :-S

Can you try and bring this back down to the level of us mere mortals by
explaining how your OR stuff would let me take a table of data in an RDBMS table
and have it appear as objects in the Management Inteferace?

I know that's horribly oversimplified, but some of us only have small brains ;-)

cheers,

Chris

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: oodb philosophics ;) was: Re: [Zope-dev] Experiments with ORMapping

2001-05-14 Thread Jeff

From: "Joachim Werner" <[EMAIL PROTECTED]>

> > This is true in the ZODB, but can be complicated by acquisition. If an
> > object can acquire itself, it can cause issues. Plus it becomes
> > difficult to know whether objects are clones or just identical
> > instances, although this can be mitigated by exposing their Python
> > instance id.
>
> Acquisition is very cool, but it sometimes really sucks ... AFAIK you can
> easily "switch it off" in your own Python products. But I am still
fighting
> with only getting private variables (i.e. not acquired ones) in DTML ...
>
>

>From DTML I have used 2 different methods for this:
1) 

 .


or
2) 

   ...


In both cases 'object' is the thing with the 'private variables' and in 2),
'attribute' is the 'variable' name.
Jeff




___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: oodb philosophics ;) was: Re: [Zope-dev] Experiments with ORMapping

2001-05-14 Thread Shane Hathaway

Joachim Werner wrote:
> 
> > Probably I'm daft because it is Friday night, but AFAIK ZODB and most
> OODB's
> > store an object only once, keyed by its object id. The rest is just
> references
> > through that oid, so objects that belong to more than one container can be
> > added to all these containers and n:m relations are implemented by having
> a
> > list of objects on both sides.
> 
> Yes, but these references (let's call them "symlinks" ...) are not in the
> standard Zope. Of course it should be easy to do these things, but except
> for this from Shane I haven't seen anything so far:
> http://www.zope.org/Members/hathawsh/Symlink/index_html

Actually I was referring to the OODB-equivalent of "hard links".  It's
only possible from Python products / external methods / filesystem
scripts / Zope core code, but here's how you do it:

ob = some_folder.some_child
some_other_folder.some_child = ob

In other words, if you just assign an object to have multiple parents
then it will "just work".  Changes to the object will appear to occur in
both places, but really they're only occurring in one place.  This
behavior is fully persistent, works over ZEO, and ZODB won't even see
there's anything unusual.

> P.S.: Shane, have you developed Symlinks any further? I think they could be
> extremely useful. I tried out the initial release and liked it, except for
> the fact that the symlinks looked EXACTLY like real ones, so they can be
> very irritating ...

I'm not sure what to do with symlinks.  How should security be applied? 
How are they most useful?

It's neat to see it works, though. :-)

Shane

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Experiments with ORMapping

2001-05-14 Thread Shane Hathaway

"Phillip J. Eby" wrote:
> 
> At 05:42 PM 5/11/01 -0400, Shane Hathaway wrote:
> >"Phillip J. Eby" wrote:
> >> I'm not quite clear on how exactly you suggest mapping from RDMBS ->
> >> ZODB.  There's a *significant* (IMHO) impedance mismatch between ZODB's
> >> arbitrarily identified variably structured single records and SQL's
> >> content-identified fixed-structure record sets.  This is what application
> >> frameworks/toolkits (such as your own DBAPI product) are needed for.
> >
> >If you implement this at the Storage level, yes, there is a major
> >mismatch.  But at the Connection level it makes a lot of sense.
> >Connection basically exposes a pile of pickles as objects; an OR mapping
> >exposes a complex schema as objects.
> >
> >I think that understanding will change the rest of your response. :-)
> >
> 
> Nope.  As far as I can see, all that does is leverage ZODB Connections as a
> crude sort of Rack that only provides a mapping-like interface for
> retrieving objects, without helping any of the higher-order needs of an
> application.  I guess if you define O-R mapping as "can you store and
> retrieve an object's properties from a database", then I guess you've
> succeeded at that point.  But if that was all my O-R apps needed, there
> would've been no reason to use the RDBMS; I could've just used ZODB and a
> BTree with less bother.

I'm telling you there's a lot more you can do with the code that makes
up ZODB.  The mapping interface is not the key to the way Connection
does its work.  OIDs don't have to be strings.  If we just use
cPersistent, cPickleCache (which is misnamed), and Transaction to
implement an OR mapping, here's what we get for free:

- A transparent, transactional database.

- Potential interoperability with everything written for ZODB.

- Robust, tested code.

- Optimizations in C.

Since you challenged me :-) I decided to put together a prototype.  What
I came up with was a database connection that sets a different _p_jar
for each persistent object.  The _p_jar is an object that manages the
storage for only one persistent object, making it possible to customize
the storage.  There was only one hurdle in this approach but I hacked
around it: the tpc_* messages get sent to each _p_jar.  Then I wrote a
very minimal test that doesn't connect to an RDBMS but stores and
retrieves simple objects in a dictionary.

OIDs are still necessary to support multithreaded invalidation
messages.  But the OIDs in the prototype are a tuple consisting of a
schema ID and a record ID.  That way the record IDs only need to be
unique within an RDBMS table (or combination of tables.)

I didn't think anything like this was possible before I got to know
Jim.  I still didn't understand when he presented the idea months ago. 
But now I see the idea really is workable.  The advantage is that it
lets us completely isolate RDBMS storage details from the application.

The next thing to do is to write a fishbowl proposal.

Shane

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: oodb philosophics ;) was: Re: [Zope-dev] Experiments with ORMapping

2001-05-14 Thread Mail List

From: "Joachim Werner" <[EMAIL PROTECTED]>

> > This is true in the ZODB, but can be complicated by acquisition. If an
> > object can acquire itself, it can cause issues. Plus it becomes
> > difficult to know whether objects are clones or just identical
> > instances, although this can be mitigated by exposing their Python
> > instance id.
>
> Acquisition is very cool, but it sometimes really sucks ... AFAIK you can
> easily "switch it off" in your own Python products. But I am still
fighting
> with only getting private variables (i.e. not acquired ones) in DTML ...
>
>From DTML I have used 2 different methods for this:
1) 
 .


or
2) 
   ...


In both cases 'object' is the thing with the 'private variables' and in 2),
'attribute' is the 'variable' name.
Jeff



___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: oodb philosophics ;) was: Re: [Zope-dev] Experiments with ORMapping

2001-05-13 Thread Joachim Werner

> Probably I'm daft because it is Friday night, but AFAIK ZODB and most
OODB's
> store an object only once, keyed by its object id. The rest is just
references
> through that oid, so objects that belong to more than one container can be
> added to all these containers and n:m relations are implemented by having
a
> list of objects on both sides.

Yes, but these references (let's call them "symlinks" ...) are not in the
standard Zope. Of course it should be easy to do these things, but except
for this from Shane I haven't seen anything so far:
http://www.zope.org/Members/hathawsh/Symlink/index_html

P.S.: Shane, have you developed Symlinks any further? I think they could be
extremely useful. I tried out the initial release and liked it, except for
the fact that the symlinks looked EXACTLY like real ones, so they can be
very irritating ...

Joachim


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: oodb philosophics ;) was: Re: [Zope-dev] Experiments with ORMapping

2001-05-13 Thread Joachim Werner

> This is true in the ZODB, but can be complicated by acquisition. If an
> object can acquire itself, it can cause issues. Plus it becomes
> difficult to know whether objects are clones or just identical
> instances, although this can be mitigated by exposing their Python
> instance id.

Acquisition is very cool, but it sometimes really sucks ... AFAIK you can
easily "switch it off" in your own Python products. But I am still fighting
with only getting private variables (i.e. not acquired ones) in DTML ...


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Experiments with ORMapping

2001-05-12 Thread Phillip J. Eby

At 05:42 PM 5/11/01 -0400, Shane Hathaway wrote:
>"Phillip J. Eby" wrote:
>> I'm not quite clear on how exactly you suggest mapping from RDMBS ->
>> ZODB.  There's a *significant* (IMHO) impedance mismatch between ZODB's
>> arbitrarily identified variably structured single records and SQL's
>> content-identified fixed-structure record sets.  This is what application
>> frameworks/toolkits (such as your own DBAPI product) are needed for.
>
>If you implement this at the Storage level, yes, there is a major
>mismatch.  But at the Connection level it makes a lot of sense. 
>Connection basically exposes a pile of pickles as objects; an OR mapping
>exposes a complex schema as objects.
>
>I think that understanding will change the rest of your response. :-)
>

Nope.  As far as I can see, all that does is leverage ZODB Connections as a
crude sort of Rack that only provides a mapping-like interface for
retrieving objects, without helping any of the higher-order needs of an
application.  I guess if you define O-R mapping as "can you store and
retrieve an object's properties from a database", then I guess you've
succeeded at that point.  But if that was all my O-R apps needed, there
would've been no reason to use the RDBMS; I could've just used ZODB and a
BTree with less bother.


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: oodb philosophics ;) was: Re: [Zope-dev] Experiments with ORMapping

2001-05-11 Thread Cees de Groot


[EMAIL PROTECTED] said:
> This is true in the ZODB, but can be complicated by acquisition.

Cheap cop-out: I was planning to avoid acquisition in my "business object" 
database (which will be a database separate from the Zope ZODB). Personally, 
I'm not very convinced of the necessity and advantages of acquisition, 
contrasted with the issues that often arise because acquisition picks a 
different thing from what the developer thought it would do. It seems OK for a 
strictly hierarchical collection of web-enabled objects, but I am thinking 
about a business model, where I don't see a lot of advantages in inflicting 
behavior onto objects through a containment hierarchy. But then, I didn't read 
the original paper so please prove me wrong.


-- 
Cees de Groot   http://www.cdegroot.com <[EMAIL PROTECTED]>
GnuPG 1024D/E0989E8B 0016 F679 F38D 5946 4ECD  1986 F303 937F E098 9E8B



___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: oodb philosophics ;) was: Re: [Zope-dev] Experiments with ORMapping

2001-05-11 Thread Casey Duncan

Cees de Groot wrote:
> 
> Joachim Werner <[EMAIL PROTECTED]> said:
> >[...]. E.g. how would
> >you handle objects beloning to more than one container? In SQL this is easy
> >(Just have a table that matches key pairs from the container table and the
> >item table). And I don't know any good way of implementing "many-to-many"
> >relations in object hierarchies. Let alone querying them efficiently.
> >
> Probably I'm daft because it is Friday night, but AFAIK ZODB and most OODB's
> store an object only once, keyed by its object id. The rest is just references
> through that oid, so objects that belong to more than one container can be
> added to all these containers and n:m relations are implemented by having a
> list of objects on both sides.

This is true in the ZODB, but can be complicated by acquisition. If an
object can acquire itself, it can cause issues. Plus it becomes
difficult to know whether objects are clones or just identical
instances, although this can be mitigated by exposing their Python
instance id.

-- 
| Casey Duncan
| Kaivo, Inc.
| [EMAIL PROTECTED]
`-->

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Experiments with ORMapping

2001-05-11 Thread Cees de Groot

Phillip J. Eby <[EMAIL PROTECTED]> said:
>For example, if I am an ISP, and I want to implement an 
>"active" flag on an "account" object, I would like changing it to 
>automatically go out and add or remove routing entries and password entries 
>on my servers when I update the record through my Zope interface (web, 
>SOAP, or whatever).  Could you make a ZODB "Storage" object that supported 
>this?  Maybe.  But from an architectural standpoint it would be rather 
>messy - akin to writing OO code with giant "switch" statements.
>

class Account(Persistent):
def disable(self):
map(lambda s: s.end(), self.subscriptions)
self.enabled = 0

I don't see a switch statement there. Granted, some jerk could come in
and code "account.enabled = 0" somewhere in his code, but that jerk
would soon be looking for other employment ;-). I do think that this
won't happen very often: instance variables in Python are very implicit,
you have to hunt the class code to find them. Instance methods are very
explicit, and therefore more likely to be found first.

(mostly everything an ISP customer orders is a kind of subscription. A
subscription has a start event, a repeat event, and an end event (or
state, or whatever you want to call it). Example webhosting: start event
is 'create Apache config', repeat event is 'send out yearly invoice'
(probably in the base class), end event is 'archive website and remove
from Apache config').

Funny you came up with this example. I run an ISP and have spent the last
couple of days thinking about just this stuff :-)



___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: oodb philosophics ;) was: Re: [Zope-dev] Experiments with ORMapping

2001-05-11 Thread Cees de Groot

Joachim Werner <[EMAIL PROTECTED]> said:
>[...]. E.g. how would
>you handle objects beloning to more than one container? In SQL this is easy
>(Just have a table that matches key pairs from the container table and the
>item table). And I don't know any good way of implementing "many-to-many"
>relations in object hierarchies. Let alone querying them efficiently.
>
Probably I'm daft because it is Friday night, but AFAIK ZODB and most OODB's
store an object only once, keyed by its object id. The rest is just references
through that oid, so objects that belong to more than one container can be
added to all these containers and n:m relations are implemented by having a
list of objects on both sides. 


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Experiments with ORMapping

2001-05-11 Thread Shane Hathaway

"Phillip J. Eby" wrote:
> 
> At 11:01 AM 5/11/01 -0400, Shane Hathaway wrote:
> >Joachim Werner wrote:
> > >
> > > The current design plans of SmartObjects are mainly based on the assumption
> > > that we will not be able to change Zope itself. This is not a dogma for us,
> > > however. I guess doing OR-mapping in the Zope core would be fine with
> > us ;-)
> >
> >FYI by replacing I don't mean changing anything in the Zope core.  I
> >mean using a different class in place of Connection, which you can do
> >just by creating a custom_zodb.py.
> 
> Hm.  So you're suggesting creation of a Storage class that returns a
> special root object which emulates the standard ZODB root
> PersistentMapping, and contains another object that emulates a folder, with
> a bunch of other foldoids that are actually tables, or something of that
> sort?  And maybe exposes some query methods ala ZCatalog?

No.  The suggestion was to return something that implements the DB
interface but doesn't use Connection.py to do it.

> I'm not quite clear on how exactly you suggest mapping from RDMBS ->
> ZODB.  There's a *significant* (IMHO) impedance mismatch between ZODB's
> arbitrarily identified variably structured single records and SQL's
> content-identified fixed-structure record sets.  This is what application
> frameworks/toolkits (such as your own DBAPI product) are needed for.

If you implement this at the Storage level, yes, there is a major
mismatch.  But at the Connection level it makes a lot of sense. 
Connection basically exposes a pile of pickles as objects; an OR mapping
exposes a complex schema as objects.

I think that understanding will change the rest of your response. :-)

Shane

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Experiments with ORMapping

2001-05-11 Thread Phillip J. Eby

At 11:01 AM 5/11/01 -0400, Shane Hathaway wrote:
>Joachim Werner wrote:
> >
> > The current design plans of SmartObjects are mainly based on the assumption
> > that we will not be able to change Zope itself. This is not a dogma for us,
> > however. I guess doing OR-mapping in the Zope core would be fine with 
> us ;-)
>
>FYI by replacing I don't mean changing anything in the Zope core.  I
>mean using a different class in place of Connection, which you can do
>just by creating a custom_zodb.py.

Hm.  So you're suggesting creation of a Storage class that returns a 
special root object which emulates the standard ZODB root 
PersistentMapping, and contains another object that emulates a folder, with 
a bunch of other foldoids that are actually tables, or something of that 
sort?  And maybe exposes some query methods ala ZCatalog?

I'm not quite clear on how exactly you suggest mapping from RDMBS -> 
ZODB.  There's a *significant* (IMHO) impedance mismatch between ZODB's 
arbitrarily identified variably structured single records and SQL's 
content-identified fixed-structure record sets.  This is what application 
frameworks/toolkits (such as your own DBAPI product) are needed for.


> > But the more important one for us is that we still believe that even with
> > ZCatalog Zope can not really do efficient "croos-tree" queries in all 
> cases.
> > A query like "Give me all users who have bought this and that product and
> > are aged 20 or above" can not be handled by the catalog, I think.
>
>Yes it can. :-)  That's not to say that it has as much flexibility as a
>SQL query, but it can do most of the things people usually do with one
>table at a time.
>
>Here's the goal I envision for OR mapping: to be able to move between
>OODBMS and RDBMS seamlessly.  It makes sense to develop on top of ZODB
>then move to an RDBMS for testing and deployment.

Unfortunately, I think that this requirement can *only* be met through a 
common API or access pattern/framework/what-have-you, be it DBAPI, 
ZPatterns, SmartObjects, or TransWarp.  The ZODB is both "too powerful" (in 
its flexibility) and "too weak" (in lack of any ZODB-level notions of 
record sets, schemas, and indexing) to be useful as a 
cross-database-platform API.  That's not to denigrate any of the value of 
ZODB itself - an explicit goal of both ZP and TW is to leverage ZODB's 
flexibility in combination with other kinds of databases.  (And, recently, 
I have been speculatively eyeing the ZODB for some mortgage-industry 
related projects which involve complex variable data structures, 
distribution requirements ala ZEO, and local data stores on 
intermittently-connected laptops.)



> > SmartObjects is more of a programming framework than just adding OR-Mapping
> > to Zope. So if we can solve the storage and query parts more efficiently by
> > just having Zope itself extended a bit, this would be very cool ...
>
>You see, I think it is not necessary to create a programming framework
>if the goal is OR mapping.  The framework is already defined, and it's
>Python / ZODB.  But SmartObjects seems to have many loosely related
>goals, making it difficult to assist.


I think the goal for SO, and all the other frameworks that have been 
created or are being created, is to *make application development 
easier*.  OR mapping is just a means to that end.  LDAP connectivity, or 
other types of non-ZODB data access are important to many of us, as 
well.  This boils down to abstraction of how "data" attributes are 
represented.  For example, if I am an ISP, and I want to implement an 
"active" flag on an "account" object, I would like changing it to 
automatically go out and add or remove routing entries and password entries 
on my servers when I update the record through my Zope interface (web, 
SOAP, or whatever).  Could you make a ZODB "Storage" object that supported 
this?  Maybe.  But from an architectural standpoint it would be rather 
messy - akin to writing OO code with giant "switch" statements.

The Java (and CORBA, actually) solution to this, is to use a "property" 
pattern, where a property is a non-real thing that actually only has 
setters and getters.  An object's users always go through these accessor 
methods, so the implementation can be anything you like.  (Notice, btw, how 
the solution is once again a framework pattern...)

ZPatterns was an effort to produce a Pythonic emulation of this concept 
that didn't need actual methods.  It unfortunately inherits some of the 
flaws of __getattr__ hooks, such as forced semi-centralization of 
implementations. TransWarp does away with that and instead introduces 
property ("feature") objects which themselves have methods, but are not 
data.  This provides a bit more extensibility than the Java approach, since 
it is conceivable one could add an "observable" interface to a property 
object and thus subscribe to it, for example.

Anyway, all I'm really trying to say is, if you want implementation 
independence, you hav

Re: oodb philosophics ;) was: Re: [Zope-dev] Experiments with ORMapping

2001-05-11 Thread David Brown

At 08:38 PM 5/11/2001 +0200, you wrote:

 >E.g. how would
 >you handle objects beloning to more than one container? In SQL this is easy
 >(Just have a table that matches key pairs from the container table and the
 >item table).

I could do the same thing with Python, creating a dictionary that does the 
mapping.  You'd have to maintain the dictionary, sure, but you'd have to 
maintain the table in a relational database as well.

 > And I don't know any good way of implementing "many-to-many"
 >relations in object hierarchies. Let alone querying them efficiently.

Well, then we need a many-to-many mapping object.  The point is that we 
could do this in Python and make the classes available, rather than 
morphing our thought processes into whatever shape the RDBMS wants.

We're just missing some functionality, but it can be written without 
rewriting the database code, because we can just store our objects 
(representing indices or mappings or whatever) directly.

The cool thing about object stores is that you can use the object modeling 
tools to do the job, and come up with specific (and more efficient) 
solutions, rather than mapping a general (and possibly slower) solution 
onto a specific problem.

dave


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Experiments with ORMapping

2001-05-11 Thread Joachim Werner

> It sounds like you are tackling a lot of things at once.  You might
> consider dividing into separate projects.

Yep, that's what is happening right now. We are still collecting "feature
requests". Then we will sort them out. A lot of the stuff will also be
useful in different contexts than we have planned, so maybe these things
should be in a different project ...

I'm sure about one thing: "SmartObjects" will be the wron name for most of
the things we are tackling ...

> > A query like "Give me all users who have bought this and that product
and
> > are aged 20 or above" can not be handled by the catalog, I think.

> Yes it can. :-)  That's not to say that it has as much flexibility as a
> SQL query, but it can do most of the things people usually do with one
> table at a time.

I am talking about MANY tables (or object relations) ... ;-)
In the example, it's users in many different sub-folders (e.g. branded
portals), products from a product catalog, etc. If you KNOW in advance that
you will need the info, you can easily store the info pairs in the Catalog,
but if you just want to do ad-hoc data mining ...

> The CMF has a solution for this, actually.  The results of a query
> include only the things you're allowed to access.

That's why I said "standard" ;-)

> > SmartObjects is more of a programming framework than just adding
OR-Mapping
> > to Zope. So if we can solve the storage and query parts more efficiently
by
> > just having Zope itself extended a bit, this would be very cool ...

> You see, I think it is not necessary to create a programming framework
> if the goal is OR mapping.  The framework is already defined, and it's
> Python / ZODB.  But SmartObjects seems to have many loosely related
> goals, making it difficult to assist.

But "Python/ZODB" is not a high-level framework. Like Zope is not the CMF.
Nobody would implement GUIs in pure C now. They are using tools for that.
And we don't want to have to implement things like HTML forms, event
handling, etc. over and over again. We want to just configure and "compile".
Of course this will be within the Python/ZODB framework, i.e. all the
additional functionality will be "support classes" or mix-in classes like
the ones Zope already offers (e.g. CatalogAware).


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: oodb philosophics ;) was: Re: [Zope-dev] Experiments with ORMapping

2001-05-11 Thread Joachim Werner

> But isn't Python a decent query language?  Isn't it nice to be able to
have
> all of the facilities of Python at your disposal when manipulating data,
> rather than hoping that whatever database you are using doesn't have a
> brain-damaged implementation of SQL?

Most of the time nobody will need SQL. And having a transparent,
transaction-aware and undoable persistent Object store is really a cool
thing. But from time to time you will hit walls. That's when the object
paradigm just does not do the job completely well any more. E.g. how would
you handle objects beloning to more than one container? In SQL this is easy
(Just have a table that matches key pairs from the container table and the
item table). And I don't know any good way of implementing "many-to-many"
relations in object hierarchies. Let alone querying them efficiently.

Joachim


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: oodb philosophics ;) was: Re: [Zope-dev] Experiments with ORMapping

2001-05-11 Thread David Brown

At 11:45 AM 5/11/2001 -0600, Casey Duncan wrote:

>One of the biggest limitations in my mind is the lack of a general query
>language for the ZODB like what you get with most OODBMS and all RDBMS.

I used to think this as well.

But isn't Python a decent query language?  Isn't it nice to be able to have 
all of the facilities of Python at your disposal when manipulating data, 
rather than hoping that whatever database you are using doesn't have a 
brain-damaged implementation of SQL?

Isn't it nice not to have to convert back and forth between SQL types and 
native types?  Isn't it nice not to have to swap in your SQL mind in the 
middle of your Python program?

Having a general query language makes it easy for people who know that 
particular general query language to write programs.  It makes it easy to 
access a bunch of different data sources, at least until the monster named 
"implementation differences" rears it's ugly head.

We've all spent years learning to make our programs interface with 
databases, learning how to jump the mental chasm between our programs and 
they way they want to manipulate data, and the way that the database wants 
to manipulate data.  Isn't it nice not to have to do that any more?

Don't get me wrong, I believe I get your point.  SQL implementations are 
getting more and more compatible.  There are OODBMS query languages 
specified.  There's no really good way of making different programming 
languages and programming environments interoperate without some sort of 
common meeting ground, like a general query language.

And perhaps I'm overdoing the response, perhaps I've gone off in a 
different direction.  I've just been thinking about this quite a bit lately.

dave


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: oodb philosophics ;) was: Re: [Zope-dev] Experiments with ORMapping

2001-05-11 Thread Casey Duncan

Cees de Groot wrote:
> 
> Shane Hathaway <[EMAIL PROTECTED]> said:
> >That's one reason ZODB is so nice.  You can write an application without
> >writing a formal schema.
> >
> One of the reasons I am seriously considering to migrate our production
> database from PostgreSQL to ZODB. I am about to implement our product
> database, and it is just too darn complex to bother maintaining SQL tables for
> it...
> 
> >Actually OracleStorage and bsddbstorage, recently released, are designed
> >to address concerns about performance and reliability, and they do an
> >excellent job at it.  And I consider ZODB as "real" an OODB as anything
> >else.  (In some ways it's the best out there IMHO.)
> >
> I heard that OracleStorage was quite a bit slower? And from what I've seen
> from FileStorage, it's a basic transaction log - what can be more reliable
> than that?
> 
> Are people using ZODB for non-Zope data? I'd be very interested to discuss
> things like emulating extents, patterns for indexing, etcetera...
> 

One of the biggest limitations in my mind is the lack of a general query
language for the ZODB like what you get with most OODBMS and all RDBMS.
ZCatalog is improving, but it is just not quite there yet.

I do feel that the ZODB is quite robust, and with the added option of
berkeley storage along with others, you have several back-end choices.

-- 
| Casey Duncan
| Kaivo, Inc.
| [EMAIL PROTECTED]
`-->

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Experiments with ORMapping

2001-05-11 Thread Shane Hathaway

Joachim Werner wrote:
> 
> Hi!
> 
> > > Do you know that there already is a project for OR-Mapping in Zope
> (actually
> > > there are two ...)?
> >
> > Yes, and I think the projects need to look into replacing parts of ZODB
> > rather than adding complexity.  ZODB has pieces that can be split apart
> > and replaced as needed, such as caching, persistence, transactions, the
> > "pickle jar", the multi-threaded connection factory, and the storage
> > layer.  I'm hoping we can achieve OR mapping by only replacing the
> > "pickle jar", i.e. Connection.py.
> 
> The current design plans of SmartObjects are mainly based on the assumption
> that we will not be able to change Zope itself. This is not a dogma for us,
> however. I guess doing OR-mapping in the Zope core would be fine with us ;-)

FYI by replacing I don't mean changing anything in the Zope core.  I
mean using a different class in place of Connection, which you can do
just by creating a custom_zodb.py.

> But I am quite sure it will not solve ALL issues SmartObjects is going to
> tackle. One of the things we want to accomplish is "better ZClasses".
> Another thing is queries: You enumerated a couple of reasons why people
> would use SQL instead of ZODB. One is integration of legacy data of course.

It sounds like you are tackling a lot of things at once.  You might
consider dividing into separate projects.

> But the more important one for us is that we still believe that even with
> ZCatalog Zope can not really do efficient "croos-tree" queries in all cases.
> A query like "Give me all users who have bought this and that product and
> are aged 20 or above" can not be handled by the catalog, I think.

Yes it can. :-)  That's not to say that it has as much flexibility as a
SQL query, but it can do most of the things people usually do with one
table at a time.

Here's the goal I envision for OR mapping: to be able to move between
OODBMS and RDBMS seamlessly.  It makes sense to develop on top of ZODB
then move to an RDBMS for testing and deployment.

> Same with
> the appointment things in our groupware project: While it is trivial for SQL
> to return me all appointments all members of my project have on a specific
> day (security is handled by the SmartObjects folders' methods in that case),
> ZODB/ZCatalog would only be able to resolve this question if either all
> appointments are stored in a central folder (what about security then?) or
> everything is cataloged (again, the standard ZCatalog does not handle
> security here, right?)

The CMF has a solution for this, actually.  The results of a query
include only the things you're allowed to access.

> What I need is an object-oriented storage (and I agree that ZODB is one of
> the best of them) that can efficiently be queried "cross-tree". One
> different approach to this topic could be a better ZCatalog (maybe one that
> stores the catalog in relational tables).

Right.  I agree an *alternate* ZCatalog would be useful.

> > > "Ours" is SmartObjects
> > > (http://demo.iuveno-net.de/iuveno/Products/SmartObjects). Also see the
> > > mailing list archive at
> > > http://imail.iuveno-net.de/pipermail/smartobjects/.
> >
> > Kapil gave Jim and I a good introduction to SmartObjects yesterday.  So
> > far, it seems a lot like ZPatterns in that it mandates a new database
> > access API rather than trying to be transparent like ZODB.
> 
> I think whether SmartObjects (in the current design) can be called
> "transparent" depends on the level. On the user level, they will be. You can
> Add them to folders, cut&paste them, upload them via FTP etc. The Zope
> security API is also used. So they are much more transparent than direct
> connections to LDAP, SQL, or to the file system (o.k. for the filesystem
> extFile does this already). Some of the API will definitely be needed: A
> SmartObject will have to be connected to databases like SQL-DAs or
> LDAPAdapter have to. So there has to be a ZMI-Plugin, and there has to be an
> API for that.
> 
> SmartObjects is more of a programming framework than just adding OR-Mapping
> to Zope. So if we can solve the storage and query parts more efficiently by
> just having Zope itself extended a bit, this would be very cool ...

You see, I think it is not necessary to create a programming framework
if the goal is OR mapping.  The framework is already defined, and it's
Python / ZODB.  But SmartObjects seems to have many loosely related
goals, making it difficult to assist.

Shane

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: oodb philosophics ;) was: Re: [Zope-dev] Experiments with ORMapping

2001-05-11 Thread Joachim Werner

> The other motivations for an RDBMS are (1) people have existing schemas
> and want Zope to access the same data as their existing apps, and they
> want it to be transparent, and (2) tables with millions of entries are
> easily stored in Zope but the perception is that the catalog isn't as
> fast as a database index.  No one has done any tests AFAIK.

Then we should do these tests. E.g. I'd like to see:

- 20 GB of Word and PDF documents stored in the ZODB and full-text +
metadata indexed in ZCatalog
- the complete [EMAIL PROTECTED] mailing list archive in the ZODB

If Zope can handle those without the help of external tools (RDBMS etc.),
I'll use it for all our Document Management ...

> That's one reason ZODB is so nice.  You can write an application without
> writing a formal schema.

Another thing (from the Slashdot article earlier this week): In Java,
changes to the object structure mean "recompile". In Zope, you can just do
them. with working "refresh" support this will even work without having to
restart the Zope process ...

> > What about using a real oodb for zope? Dont remember any particular
> > product name, but I heard something.
>
> Actually OracleStorage and bsddbstorage, recently released, are designed
> to address concerns about performance and reliability, and they do an
> excellent job at it.  And I consider ZODB as "real" an OODB as anything
> else.  (In some ways it's the best out there IMHO.)

Agreed. ZODB has a much longer proven history of success than most other
OODBs.



___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Experiments with ORMapping

2001-05-11 Thread Joachim Werner

> > > But storage of binary pickles was never the intention anyway.  I
created
> > > a little interface that would allow you to store different classes in
> > > different PostgreSQL tables.  Before I got to implementing anything,
> >
> > Is this much like the ZPatterns approach?
>
> Which part of ZPatterns are you referring to?  The idea is to
> (generally) put all instances of a class in a certain table.  But the
> implementation details having nothing in common with ZPatterns.

That's the basic approach for SmartObjects, too.

> > Or do we want to make some automatism to get tables created and
destroyed
> > according to ZClasses needs?

SmartObjects would also do that. But not with the existing ZClasses. There
will be something similar that you can use to define the schemas and that
then creates the classes and the tables on the fly.

> I was thinking there would be a default table where everything gets
> stored by default.  A programmer then tells the ORMapping about specific
> classes and how to store them.




___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Experiments with ORMapping

2001-05-11 Thread Joachim Werner

> So, if I had a.n.other SQL table containing some attributes, how would you
see
> that mappign to Zope objects?
>
> What meta_type would they be?
>
> Where would they show up?

That's why we need an API and can't just do everything transparently ;-)

We have the conept of "SmartAttributeSheets" (names can change ;-)) that can
be added to any object or class and map to certain SQL or LDAP sources.



___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Experiments with ORMapping

2001-05-11 Thread Joachim Werner

Hi!

> > Do you know that there already is a project for OR-Mapping in Zope
(actually
> > there are two ...)?
>
> Yes, and I think the projects need to look into replacing parts of ZODB
> rather than adding complexity.  ZODB has pieces that can be split apart
> and replaced as needed, such as caching, persistence, transactions, the
> "pickle jar", the multi-threaded connection factory, and the storage
> layer.  I'm hoping we can achieve OR mapping by only replacing the
> "pickle jar", i.e. Connection.py.

The current design plans of SmartObjects are mainly based on the assumption
that we will not be able to change Zope itself. This is not a dogma for us,
however. I guess doing OR-mapping in the Zope core would be fine with us ;-)

But I am quite sure it will not solve ALL issues SmartObjects is going to
tackle. One of the things we want to accomplish is "better ZClasses".
Another thing is queries: You enumerated a couple of reasons why people
would use SQL instead of ZODB. One is integration of legacy data of course.
But the more important one for us is that we still believe that even with
ZCatalog Zope can not really do efficient "croos-tree" queries in all cases.
A query like "Give me all users who have bought this and that product and
are aged 20 or above" can not be handled by the catalog, I think. Same with
the appointment things in our groupware project: While it is trivial for SQL
to return me all appointments all members of my project have on a specific
day (security is handled by the SmartObjects folders' methods in that case),
ZODB/ZCatalog would only be able to resolve this question if either all
appointments are stored in a central folder (what about security then?) or
everything is cataloged (again, the standard ZCatalog does not handle
security here, right?)

What I need is an object-oriented storage (and I agree that ZODB is one of
the best of them) that can efficiently be queried "cross-tree". One
different approach to this topic could be a better ZCatalog (maybe one that
stores the catalog in relational tables).

> > "Ours" is SmartObjects
> > (http://demo.iuveno-net.de/iuveno/Products/SmartObjects). Also see the
> > mailing list archive at
> > http://imail.iuveno-net.de/pipermail/smartobjects/.
>
> Kapil gave Jim and I a good introduction to SmartObjects yesterday.  So
> far, it seems a lot like ZPatterns in that it mandates a new database
> access API rather than trying to be transparent like ZODB.

I think whether SmartObjects (in the current design) can be called
"transparent" depends on the level. On the user level, they will be. You can
Add them to folders, cut&paste them, upload them via FTP etc. The Zope
security API is also used. So they are much more transparent than direct
connections to LDAP, SQL, or to the file system (o.k. for the filesystem
extFile does this already). Some of the API will definitely be needed: A
SmartObject will have to be connected to databases like SQL-DAs or
LDAPAdapter have to. So there has to be a ZMI-Plugin, and there has to be an
API for that.

SmartObjects is more of a programming framework than just adding OR-Mapping
to Zope. So if we can solve the storage and query parts more efficiently by
just having Zope itself extended a bit, this would be very cool ...

Cheers,
Joachim



___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Experiments with ORMapping

2001-05-11 Thread Chris Withers

Shane Hathaway wrote:
> 
> Kapil gave Jim and I a good introduction to SmartObjects yesterday.  So
> far, it seems a lot like ZPatterns in that it mandates a new database
> access API rather than trying to be transparent like ZODB.
> 
> > The other is TransWarp, which has a slightly different focus, but Phillip
> > Eby is also on our list, so we won't duplicate efforts.
> 
> TransWarp doesn't try to be transparent either AFAICT...

doing this transparently would be unbelievably cool :-)

So, if I had a.n.other SQL table containing some attributes, how would you see
that mappign to Zope objects?

What meta_type would they be?

Where would they show up?

cheers,

Chris

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: oodb philosophics ;) was: Re: [Zope-dev] Experiments with ORMapping

2001-05-10 Thread Cees de Groot

Shane Hathaway <[EMAIL PROTECTED]> said:
>That's one reason ZODB is so nice.  You can write an application without
>writing a formal schema.
>
One of the reasons I am seriously considering to migrate our production
database from PostgreSQL to ZODB. I am about to implement our product
database, and it is just too darn complex to bother maintaining SQL tables for
it...

>Actually OracleStorage and bsddbstorage, recently released, are designed
>to address concerns about performance and reliability, and they do an
>excellent job at it.  And I consider ZODB as "real" an OODB as anything
>else.  (In some ways it's the best out there IMHO.)
>
I heard that OracleStorage was quite a bit slower? And from what I've seen
from FileStorage, it's a basic transaction log - what can be more reliable
than that?

Are people using ZODB for non-Zope data? I'd be very interested to discuss
things like emulating extents, patterns for indexing, etcetera...



___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Experiments with ORMapping

2001-05-10 Thread Shane Hathaway

Joachim Werner wrote:
> 
> > You know, it might be possible to get a team together to implement
> > this.  How many out there would be interested in pursuing it further?
> > IMHO it's not as much work as it sounds at first.  Zope being so
> > object-oriented, you really can replace one of its most fundamental
> > assumptions (an OODBMS) with something else (an RDBMS) without a huge
> > effort.
> 
> Do you know that there already is a project for OR-Mapping in Zope (actually
> there are two ...)?

Yes, and I think the projects need to look into replacing parts of ZODB
rather than adding complexity.  ZODB has pieces that can be split apart
and replaced as needed, such as caching, persistence, transactions, the
"pickle jar", the multi-threaded connection factory, and the storage
layer.  I'm hoping we can achieve OR mapping by only replacing the
"pickle jar", i.e. Connection.py.

> "Ours" is SmartObjects
> (http://demo.iuveno-net.de/iuveno/Products/SmartObjects). Also see the
> mailing list archive at
> http://imail.iuveno-net.de/pipermail/smartobjects/.

Kapil gave Jim and I a good introduction to SmartObjects yesterday.  So
far, it seems a lot like ZPatterns in that it mandates a new database
access API rather than trying to be transparent like ZODB.

> The other is TransWarp, which has a slightly different focus, but Phillip
> Eby is also on our list, so we won't duplicate efforts.

TransWarp doesn't try to be transparent either AFAICT...

Shane

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Experiments with ORMapping

2001-05-10 Thread Joachim Werner

> You know, it might be possible to get a team together to implement
> this.  How many out there would be interested in pursuing it further?
> IMHO it's not as much work as it sounds at first.  Zope being so
> object-oriented, you really can replace one of its most fundamental
> assumptions (an OODBMS) with something else (an RDBMS) without a huge
> effort.

Hi Shane!

Do you know that there already is a project for OR-Mapping in Zope (actually
there are two ...)?

"Ours" is SmartObjects
(http://demo.iuveno-net.de/iuveno/Products/SmartObjects). Also see the
mailing list archive at
http://imail.iuveno-net.de/pipermail/smartobjects/.

The other is TransWarp, which has a slightly different focus, but Phillip
Eby is also on our list, so we won't duplicate efforts.

Cheers

Joachim


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: oodb philosophics ;) was: Re: [Zope-dev] Experiments with ORMapping

2001-05-10 Thread Casey Duncan

Tino Wildenhain wrote:
> 
> Hi shane,
> 
> I think the motivation people want an RDBMS storage beneth zodb is
> because they understand RDBMSes these days are performant, relieable
> and can quiete easy maintained.
> 
> I've seen Java implementations using this approach to achive persistens
> using as example "Powertier[tm]" to explicit map oop data to an RDBMS.
> I didnt like it because you have to map your objects each time you create
> a class, keep in mind not to infere with others etc...
> 
> Would it not be better to improve the abilities of the Filestorage
> to handle updates better? May be most of the storage system in C?
> With logfiles like modern RDBMSes use to incorporate fast changes?
> However, to avoid pickling/unpickling and may be to update on
> attribute-change, we need the approach you mentioned.
> 
> What about using a real oodb for zope? Dont remember any particular
> product name, but I heard something.
> 
> Regards
> Tino Wildenhain
> 

It would certainly be an interseting exercise to put Matisse or
Objectivity behind Zope as ZODB storage, however I think there will
always be kludgeyness because features of Zope wont directly map (like
versions).

I think the Berkeley storage option will eventually prove to be the
ticket. Probably sooner than later.

How about XML storage! 8^) You think startup times are slow now...

-- 
| Casey Duncan
| Kaivo, Inc.
| [EMAIL PROTECTED]
`-->

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: oodb philosophics ;) was: Re: [Zope-dev] Experiments with ORMapping

2001-05-10 Thread Shane Hathaway

Tino Wildenhain wrote:
> I think the motivation people want an RDBMS storage beneth zodb is
> because they understand RDBMSes these days are performant, relieable
> and can quiete easy maintained.

The other motivations for an RDBMS are (1) people have existing schemas
and want Zope to access the same data as their existing apps, and they
want it to be transparent, and (2) tables with millions of entries are
easily stored in Zope but the perception is that the catalog isn't as
fast as a database index.  No one has done any tests AFAIK.

> I've seen Java implementations using this approach to achive persistens
> using as example "Powertier[tm]" to explicit map oop data to an RDBMS.
> I didnt like it because you have to map your objects each time you create
> a class, keep in mind not to infere with others etc...

That's one reason ZODB is so nice.  You can write an application without
writing a formal schema.

> Would it not be better to improve the abilities of the Filestorage
> to handle updates better? May be most of the storage system in C?
> With logfiles like modern RDBMSes use to incorporate fast changes?
> However, to avoid pickling/unpickling and may be to update on
> attribute-change, we need the approach you mentioned.
> 
> What about using a real oodb for zope? Dont remember any particular
> product name, but I heard something.

Actually OracleStorage and bsddbstorage, recently released, are designed
to address concerns about performance and reliability, and they do an
excellent job at it.  And I consider ZODB as "real" an OODB as anything
else.  (In some ways it's the best out there IMHO.)

Shane

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



oodb philosophics ;) was: Re: [Zope-dev] Experiments with ORMapping

2001-05-10 Thread Tino Wildenhain

Hi shane,

I think the motivation people want an RDBMS storage beneth zodb is
because they understand RDBMSes these days are performant, relieable
and can quiete easy maintained.

I've seen Java implementations using this approach to achive persistens
using as example "Powertier[tm]" to explicit map oop data to an RDBMS.
I didnt like it because you have to map your objects each time you create
a class, keep in mind not to infere with others etc...

Would it not be better to improve the abilities of the Filestorage
to handle updates better? May be most of the storage system in C?
With logfiles like modern RDBMSes use to incorporate fast changes?
However, to avoid pickling/unpickling and may be to update on
attribute-change, we need the approach you mentioned.

What about using a real oodb for zope? Dont remember any particular
product name, but I heard something.

Regards
Tino Wildenhain


--On Donnerstag, 10. Mai 2001 12:39 -0400 Shane Hathaway 
<[EMAIL PROTECTED]> wrote:

> Tino Wildenhain wrote:
>> > But storage of binary pickles was never the intention anyway.  I
>> > created a little interface that would allow you to store different
>> > classes in different PostgreSQL tables.  Before I got to implementing
>> > anything,
>>
>> Is this much like the ZPatterns approach?
>
> Which part of ZPatterns are you referring to?  The idea is to
> (generally) put all instances of a class in a certain table.  But the
> implementation details having nothing in common with ZPatterns.
>
>> Or do we want to make some automatism to get tables created and destroyed
>> according to ZClasses needs?
>
> I was thinking there would be a default table where everything gets
> stored by default.  A programmer then tells the ORMapping about specific
> classes and how to store them.
>
>> > So, if we have context, the relational mapping tree can work.  It would
>> > turn Zope into a purely relational application server, which a lot of
>> > folks apparently want.  ;-)
>>
>> Oh, do they? ;)
>> Me dont :-)
>
> You know, it might be possible to get a team together to implement
> this.  How many out there would be interested in pursuing it further?
> IMHO it's not as much work as it sounds at first.  Zope being so
> object-oriented, you really can replace one of its most fundamental
> assumptions (an OODBMS) with something else (an RDBMS) without a huge
> effort.
>
> Shane





___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Experiments with ORMapping

2001-05-10 Thread Shane Hathaway

Tino Wildenhain wrote:
> > But storage of binary pickles was never the intention anyway.  I created
> > a little interface that would allow you to store different classes in
> > different PostgreSQL tables.  Before I got to implementing anything,
> 
> Is this much like the ZPatterns approach?

Which part of ZPatterns are you referring to?  The idea is to
(generally) put all instances of a class in a certain table.  But the
implementation details having nothing in common with ZPatterns.

> Or do we want to make some automatism to get tables created and destroyed
> according to ZClasses needs?

I was thinking there would be a default table where everything gets
stored by default.  A programmer then tells the ORMapping about specific
classes and how to store them.

> > So, if we have context, the relational mapping tree can work.  It would
> > turn Zope into a purely relational application server, which a lot of
> > folks apparently want.  ;-)
> 
> Oh, do they? ;)
> Me dont :-)

You know, it might be possible to get a team together to implement
this.  How many out there would be interested in pursuing it further? 
IMHO it's not as much work as it sounds at first.  Zope being so
object-oriented, you really can replace one of its most fundamental
assumptions (an OODBMS) with something else (an RDBMS) without a huge
effort.

Shane

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Experiments with ORMapping

2001-05-10 Thread Tino Wildenhain

Hi Shane,

--On Donnerstag, 10. Mai 2001 11:32 -0400 Shane Hathaway 
<[EMAIL PROTECTED]> wrote:

...
> But storage of binary pickles was never the intention anyway.  I created
> a little interface that would allow you to store different classes in
> different PostgreSQL tables.  Before I got to implementing anything,

Is this much like the ZPatterns approach?
Or do we want to make some automatism to get tables created and destroyed
according to ZClasses needs?

> though, I had to move to another project.  But the file "sketch" shows
> (I hope) what I had in mind for making this work: programmers would call
> a bunch of functions that would put together a relational mapping tree.
>
> Now, in our discussion yesterday we decided ORStorage wasn't the right
> way to achieve relational mapping because there is no way for the
> database storage layer to have any context just given an OID.  If we
> instead consider writing a replacement for the stuff that's mainly in
> Connection.py, life gets a lot simpler.  At this level it *is* possible
> to know the parent OID of an object, though the current codebase does
> not use this opportunity.  Also, we wouldn't have to pickle and unpickle
> unnecessarily.
>
> So, if we have context, the relational mapping tree can work.  It would
> turn Zope into a purely relational application server, which a lot of
> folks apparently want.  ;-)

Oh, do they? ;)
Me dont :-)

Regards
Tino

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



[Zope-dev] Experiments with ORMapping

2001-05-10 Thread Shane Hathaway

Kapil (and others),

Although very sketchy and I can't guarantee anything works, my
experiments with object-relational mapping in Zope are found at
http://www.zope.org/Members/hathawsh/ormapping.tar.gz .  If ORMapping.py
is in the ZODB directory, you can use the following custom_zodb.py to
run Zope on top of Postgres using PoPy:

import ZODB
from ZODB.ORStorage import ORStorage
import PoPy
 
connection = PoPy.connect('user=zodb dbname=zodb')
storage = ORStorage('pgsql!', connection, PoPy, create=1)
DB = ZODB.DB(storage)

The initial approach turned out to be rather problematic, since by
default everything is stored in Postgres as a large binary object.  Zope
creates literally thousands of objects when starting for the first time
and a few hundred each time thereafter.  Postgres does not deal well
with large binary objects so start time is something on the order of 10
minutes this way and hard drive space gets eaten up faster than Jim can
code. :-)

But storage of binary pickles was never the intention anyway.  I created
a little interface that would allow you to store different classes in
different PostgreSQL tables.  Before I got to implementing anything,
though, I had to move to another project.  But the file "sketch" shows
(I hope) what I had in mind for making this work: programmers would call
a bunch of functions that would put together a relational mapping tree.

Now, in our discussion yesterday we decided ORStorage wasn't the right
way to achieve relational mapping because there is no way for the
database storage layer to have any context just given an OID.  If we
instead consider writing a replacement for the stuff that's mainly in
Connection.py, life gets a lot simpler.  At this level it *is* possible
to know the parent OID of an object, though the current codebase does
not use this opportunity.  Also, we wouldn't have to pickle and unpickle
unnecessarily.

So, if we have context, the relational mapping tree can work.  It would
turn Zope into a purely relational application server, which a lot of
folks apparently want.  ;-)

Shane

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )