[AL] As well as an ETA for Btrees, in the absence of ZPatterns documentation, could somebody please post URLs for specific references to further information about Btrees in particular and Zpatterns in general, - eg email list archives and other Wickis particular thread URLs etc. I've been studying the ZPatterns Wicki and code with a view to following up on Kevin's earlier email reference to it which I posted to Zcommerce recently. Didn't notice any explicit reference to Btree folders within ZPatterns. As far as I can make out, Kevin is saying, and appears to be right, that by designing for "Zpatterns" it should be feasible use either ZODB or an RDBM without having to rewrite the whole application, and also feasible to rewrite just the parts that are affected by different facilities on different RDBMs to support more than one RDBM option. (Of course if a Zope application is NOT designed from the start to do that, but simply assumes the ZODB will be the only data source, this would not apply. Unfortunately the two currently available Zope ecommerce products appear to have taken that route). However I got a strong impression ZPatterns may not be as useful as I thought, in a recent exchange with Chris McDonough in an OpenACS bboard (attached, with link to discussion). Any comments and references from Kevin and other people more familiar with ZPatterns would be appreciated. BTW While Btrees might make ZODB usable with very large folders I can't see how it would make it better than using an RDBM in that situation (hopefully under Zope, using ZPatterns). Once you have a lot more data than fits in RAM you are likely to also have a higher ratio of writes to reads than ZODB is "aggressively optimized" for, and need an application design that explicitly optimizes for relational operations on subsets of data rather than navigating from one linked object to another as OODBMs are designed to do. Using an OODBM like ZODB would mean a full disk seek for every item, which RDBMs are designed to avoid. Likewise if you need concurrent external access to the same data via ODBC I don't see how ZODB Btrees would help (eg for ecommerce links to General Ledger and fulfilment etc). What worries me is that in the absence of actual working examples of how to handle RDBMs with ZPatterns, I have had to recommend that someone needing an RDBM based application (Ecommerce) use the OpenACS ecommerce module and just kludge a way to also use Zope for other things in parallel. I'll be discussing the specific implications for ecommerce in Zcommerce, but perhaps here is better for info on ZPatterns and RDBMs generally? Also, although still not sufficiently well informed, I have a strong feeling that the features of Postgresql as an ORDBM as opposed to an ODBM or an RDBM are particularly relevant to Zope. As well as a normal external Python API, Postgresql has the capability to run scripting code as well as C functions inside the database engine as a stored procedure language and also for dynamically loaded user defined data types and comparison operators etc. It is a true "Object" Relational DBMS rather than a conventional RDBMs. Already implemented for Tcl. If some Zope/Python code was put inside it, designed to work with attribute providers and especially indexing agents and rule agents for ZPatterns, it strikes me that the combination could be incredibly powerful. Has there been any discussion of that, or any thoughts now? -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of Jimmie Houchin Sent: Sunday, June 18, 2000 7:12 AM To: Kevin Dangoor Cc: Zope Mailing List Subject: Re: [Zope] Big folders [JH] Thanks, this is great info. I was planning on developing with ZPatterns but haven't downloaded or began yet. I wasn't aware that a Rack could do that. Cool. I don't mind going to a RDBMS if necessary, but if possible I like sticking with Zope/ZODB. Does anyone have an eta on BTrees for ZODB? Jimmie Houchin Kevin Dangoor wrote: > > [snip] [JH] > > My databases will have millions of objects in what would/will be single > > table/folders. From what I understand ZODB doesn't currently do big > > folders well. > > > > So I guess my question is, should I continue with the SQL and the RDBMS > > or is there a solution that I don't know about or understand with ZODB? > [KD] > As Shane mentioned, there's a BTreeFolder in the works. Additionally, you > can use the "Rack" from ZPatterns. If you design for a Rack, you can switch > to an RDBMS later on without having to change your application code. But, > you might not need to switch to an RDBMS, because the current Rack > implementation can store objects using BTrees within the ZODB. So, > theoretically, you can have great big piles of objects within a single Rack. > > Unfortunately, ZPatterns is still very new and there is not a lot of > documentation. But, when you start playing with it, things become a bit > clearer. > > Kevin >
Chris McDonough ([EMAIL PROTECTED]) responded to a message you requested notification for in the OpenACS compared to other toolkits and technologies bboard: Subject: Response to Article on ACS x Zope To answer some of Albert's q's... The ZPatterns stuff is the brainchild of Phillip Eby, a developer at Verio (the hosting provider). It (among other things) is a mechanism to provide medium-independent "propertysheet" support. "Propertysheets" in Zope are rectangular sets of typed data that can be attached to Zope objects arbitrarily, and as such lend themselves very well to storage in an RDB table. And I think this was probably what Philip had in mind in providing Dynamic Data Objects. Although to be honest, I might be totally wrong... I can't really tell what the intent is now. I've heard it's a good set of ideas, but personally I'm stumped most of the time when I read his stuff because he names things very opaquely and seems to explain things in such a way that you *do* need to community college C++ degree to begin to grasp it. I think it's one of those projects that needs to gain some traction and obtain a critical mass of real-world examples in order to make its ultimate intent clear to mortals like me. Until then, I can't speak much for it, although I do recommend you continue looking at it. (Maybe you can write some code using it that would explain it to me!) ZPatterns is not part of the Zope distribution. I think Ben had in mind at least in his most recent posting an abstraction layer that allowed RDB-savvy people to write SQL-like queries that would automagically be translated into their equivalent optimized Oracle- or Postgres-specific SQL dialect at runtime. I don't think anybody in Zopeland has the intent of writing such a beast. You're probably right in assuming that the ZPatterns stuff is about as close as it gets to wanting to do something like that. But Zope propertysheets are generally by nature pretty small fixed sets of data (maybe 5 - 12 properties per object). You would need to declare some sort of table structure for a particular object type, create the tables to store the propertysheet data for those kinds of objects, then do the right voodoo to tell the objects to obtain their properties out of SQL instead of out of the ZODB. This could probably be abstracted in such a way that people don't need to write any SQL, but it would never approach the sort of complexity that I think Ben is talking about because it would probably practically necessitate an approach something along the lines of defining a table to hold propertysheet data for each different object type. If you wanted to do ad hoc queries against an existing set of data in multiple tables (ones that might require unions, outer joins, or insert selects) would need to be (just as painfully) manually constructed in the SQL dialect of your choice. Personally, I'd probably try to keep it simple and create views on existing tables that would let me operate as if I were doing so on a single table. Somehow I get the feeling that in this conversation we're all talking an entirely different language than one another. I assume that Ben's concerned about complicated SQL abstraction between Oracle and Postgres because he wants to get rid of Oracle dependencies in ACS. I think Albert's talking about being able to abstract the SQL out enough to not have to think about it at all when constructing an application. I'm probably just plain not thinking about it at all. I looked at Ben's sql-abstraction page... I don't know any tcl, but I think I get the gist. It would seem to me to make sense to abstract out both 1) the access mechanism for querying and the 2) resultset that comes back from the query. In Zope we do the first part by creating a "method" that contains the actual query (which can be as DB-specific or generic as you like), connecting it to a database connection (a pool of database handles for a specific DB type). You can then pass in arguments to the method that have counterpart template definitions within the query itself (ala CUSTID = <dtml-sqlvar abc>), and the method passes the query in to the DB and obtains its results. The second part (returning the results of the query to the caller) is handled in Zope by wrapping the resultset in a class instance that lets us process those results generically. It would probably be good to do something like this in ACS by defining a datatype that could hold arbitrary common elements that could be processed in tcl code generically. Though it's not a trivial engineering task, it seems like the biggest problem isn't implementing it, but encouraging people to use it. From what I understand there's a large body of existing code that does not use any sort of abstraction like this. Convincing folks of the usefulness of writing against your interface instead of yanking the stuff right out of the tables in their code is probably the hardest task. ----------------- To post a response, come back to the bulletin board at http://openacs.org/bboard/q-and-a.tcl?topic_id=8&topic=OpenACS%20compared%20 to%20other%20toolkits%20and%20technologies ------------- If you are no longer interested in this thread, simply go to the following URL and you will no longer get these notifications: http://openacs.org/bboard/shut-up.tcl?row_id=1488768 ------------- Note: this message was sent by a robot.
Albert Langer ([EMAIL PROTECTED]) responded to a message you requested notification for in the OpenACS compared to other toolkits and technologies bboard: Subject: Response to Article on ACS x Zope [CM] There is no SQL abstraction layer in Zope like Ben's talking about. You still need to write DB-specific SQL to gain results out of a query. To my knowledge, the only system that abstracts this layer out well is Apple's WebObjects which I believe can use Oracle, Sybase and Informix as its object persistence layer without requring that you write any SQL. We don't do this. We instead use the ZODB as our object persistence layer, and we let people connect to RDB's and use that RDB's specific set of SQL commands. To get anything useful out of the RDB's, you need to write system-specific SQL. [AL] My understanding is that Ben is talking about something quite different from WebObjects. To clarify that, I have posted some notes about Web Objects in a separate topic, "Apple WebObjects", on this bboard: http://openacs.org/bboard/q-and-a-fetch-msg.tcl?msg_id=0000Hi&topic_id=8&top ic=OpenACS%20compared%20to%20other%20toolkits%20and%20technologies Can we all agree that something like WebObjects is beyond the scope of what OpenACS is after for data abstraction and beyond the scope of what Zope intends to provide? >From my perspective the use of ZODB as an object persistent layer is fine for many things but completely unacceptable for Ecommerce OLTP transactions. For those you need not only to get things out of RDBMs, but also put them into RDBMs and the application is really an RDBM application. I don't buy the idea that Ecommerce based on ZODB alone without an RDBM is viable, for reasons explained in 1) and 2) of: http://lists.codeit.com/pipermail/zcommerce/2000-June/000257.html BTW If you are aware of anybody else attempting Ecommerce OLTP with an OODBMs please provide a reference. Otherwise please confirm that Zcommerce efforts to do so using ZODB without an RDBM should be considered "experimental". Despite that, I still believe Zope could be useful together with an RDBM based Ecommerce module and I am interested in whether it is possible for the existing ACS Ecommerce module to be made more "stand alone" and used with both OpenACS and Zope, or whether development would inevitably need to "fork". I claimed that: For access to RDBMs databases via SQL, Zope provides pretty much exactly the sort of data abstraction facilities that Ben was talking about in: http://openacs.org/doc/sql-abstraction.html Based on that view I am proposing that OpenACS people and Zope people should get together in work on data abstraction. Your comments below, and those from Jimmie Houchin, seem to indicate that I'm wrong about that. Given that you know far more about Zope than I do, I'll have to accept your view. But before abandoning the idea I'd like to get it clear whether we are really talking about the same thing. [CM] Zope does abstract out the ability to process results of SQL queries. If you've got a Zope system with connections to different databases (for example if you're grabbing stuff out of an Oracle DB and a Postgres DB within the same Zope instance), you create database connections to the two DB's, you create "SQL methods" for each query for each DB, and then in Python or DTML you pass arguments in to the "SQL method" that houses each query, and you can process its results generically. You needn't worry about where the data came from. Here's an example: SQL method "getstatus" query: SELECT status FROM customer WHERE id = <dtml-sqlvar custid> DTML method to process the results from this query: Customer status: <dtml-var "getstatus(custid=1)"> The DTML method passes the value of custid in to the SQL method, gets the result back, and displays it. This is the extent of Zope's SQL abstraction. You can process query results independent of the data source by creating a "database connection" to a specific DB (Oracle, Postgres, MySQL, Solid, ODBC, and Sybase) then creating "SQL methods" against these database connections that accept arguments and return results. The content of the SQL methods is necessarily database specific. [AL] Above is my understanding of what "Z SQL" offers. But I thought the "Dynamic Data Objects" were intended to provide rather more. Could you please comment more specifically on the following excerpt from my previous posting and explain what I have misunderstood: ************** The next level currently requires some vOOdOO to understand "ZPatterns". See especially the section on "Dynamic Data Objects" in the contents map at: http://www.zope.org/Members/pje/Wikis/ZPatterns/HomePage/map Suffice to say that it is NOT one of the three options that are possible without "very OO database OO" but a fourth option made possible by OO. Yes, you still need RDBM specific (or site specific) SQL for optimization, because there is NO WAY that can be avoided. But it hides this behind umpteen layers of framework code so that it is completely insulated from the application domain queries and updates and UI templates written by non-SQL developers. This makes life much harder for people implementing the framework (ie core Zope developers), just as life is hard for people writing the internals of Postgresql. But it makes life much easier for people customizing UIs, people customizing queries and updates based on application domain models and for people optimizing RDBMs engine access, because they can all keep out of each other's way. (If it's the same bunny doing them all, at least they can think about just one thing at a time). *************** What I had in mind is that Ben's document refers to a simple template for inserting parameters, into SQL much like what "Z SQL" provides, but with tags like {Oracle, Postgresql} which indicate whether a particular SQL statement is suitable for use with a particular RDBM. This would simplify maintenance of multiple ports by reusing whatever can be reused and providing sufficient isolation from the rest of the application for SQL development and Tcl development to be less tightly coupled. It seemed to me that although the "ZPatterns" framework is currently used only for multiple login authentication (eg from LDAP directory or RDBM instead of Zope folders in the ZODB), it is intended to be far more widely applicable and could be used to provide data abstraction of the sort Ben is after, by people who understand the problems of porting between different RDBMs as well as for providing application Domain Models, Roles and facades for people using the ZODB. ----------------- To post a response, come back to the bulletin board at http://openacs.org/bboard/q-and-a.tcl?topic_id=8&topic=OpenACS%20compared%20 to%20other%20toolkits%20and%20technologies ------------- If you are no longer interested in this thread, simply go to the following URL and you will no longer get these notifications: http://openacs.org/bboard/shut-up.tcl?row_id=1488768 ------------- Note: this message was sent by a robot.