Re: [ZODB-Dev] Making ZODB / ZEO faster

2009-12-04 Thread Shane Hathaway
Jim Fulton wrote: > I find this a bit confusing. For the warm numbers, It looks like ZEO didn't > utilize a persistent cache, which explains why the ZEO numbers are the > same for hot and cold. Is that right? Yes. It is currently difficult to set up ZEO caches, which I consider an issue with th

Re: [ZODB-Dev] Making ZODB / ZEO faster

2009-12-04 Thread Jim Fulton
On Fri, Dec 4, 2009 at 3:41 PM, Shane Hathaway wrote: > Jim Fulton wrote: >> >> On Fri, Dec 4, 2009 at 3:07 PM, Shane Hathaway >> wrote: >>> >>> http://shane.willowrise.com/archives/relstorage-1-4-0b1-and-zodbshootout/ >> >> I won't take the time now to analyze the new test, although I will ask >

Re: [ZODB-Dev] Making ZODB / ZEO faster

2009-12-04 Thread Jim Fulton
On Fri, Dec 4, 2009 at 3:41 PM, Shane Hathaway wrote: > Jim Fulton wrote: >> >> On Fri, Dec 4, 2009 at 3:07 PM, Shane Hathaway >> wrote: >>> >>> http://shane.willowrise.com/archives/relstorage-1-4-0b1-and-zodbshootout/ >> >> I won't take the time now to analyze the new test, although I will ask >

Re: [ZODB-Dev] Making ZODB / ZEO faster

2009-12-04 Thread Shane Hathaway
Jim Fulton wrote: > On Fri, Dec 4, 2009 at 3:07 PM, Shane Hathaway wrote: >> http://shane.willowrise.com/archives/relstorage-1-4-0b1-and-zodbshootout/ > > I won't take the time now to analyze the new test, although I will ask > a couple of questions: > > First, in your results, you show cold, wa

Re: [ZODB-Dev] Making ZODB / ZEO faster

2009-12-04 Thread Shane Hathaway
Erik Dahl wrote: > I haven't dove into the relstorage yet but have heard it will perform > better. Not sure I understand why though. Isn't it just putting > pickles into a single table with an index on the oid? (or oid / serial). Yes. In theory, ZEO should be about the same speed, but all t

Re: [ZODB-Dev] Making ZODB / ZEO faster

2009-12-04 Thread Jim Fulton
On Fri, Dec 4, 2009 at 3:07 PM, Shane Hathaway wrote: > Jim Fulton wrote: >> >> On Fri, Dec 4, 2009 at 11:31 AM, Erik Dahl wrote: >> ... >>> >>> I haven't dove into the relstorage yet but have heard it will perform >>> better. >> >> It doesn't.  See: >> >>  https://mail.zope.org/pipermail/zodb-de

Re: [ZODB-Dev] Making ZODB / ZEO faster

2009-12-04 Thread Shane Hathaway
Jim Fulton wrote: > On Fri, Dec 4, 2009 at 11:31 AM, Erik Dahl wrote: > ... >> I haven't dove into the relstorage yet but have heard it will perform >> better. > > It doesn't. See: > > https://mail.zope.org/pipermail/zodb-dev/2009-October/012758.html Since you posted those numbers, I have wo

Re: [ZODB-Dev] Making ZODB / ZEO faster

2009-12-04 Thread Chris Withers
Jim Fulton wrote: > On Fri, Dec 4, 2009 at 11:31 AM, Erik Dahl wrote: > ... >> I haven't dove into the relstorage yet but have heard it will perform >> better. > > It doesn't. See: > > https://mail.zope.org/pipermail/zodb-dev/2009-October/012758.html Jim, That's a little sweeping and some m

Re: [ZODB-Dev] Making ZODB / ZEO faster

2009-12-04 Thread Jim Fulton
On Fri, Dec 4, 2009 at 9:41 AM, Erik Dahl wrote: > Guys, > > We have a product written in python using ZODB/ZEO and I would like to > improve the speed of database in general.  Things that I have seen > that I would like to improve some I understand and some not. How have you "seen" these? Do you

Re: [ZODB-Dev] Making ZODB / ZEO faster

2009-12-04 Thread Jim Fulton
On Fri, Dec 4, 2009 at 9:41 AM, Erik Dahl wrote: > Guys, > > We have a product written in python using ZODB/ZEO and I would like to > improve the speed of database in general.  Things that I have seen > that I would like to improve some I understand and some not. How have you "seen" these? Do you

Re: [ZODB-Dev] Making ZODB / ZEO faster

2009-12-04 Thread Jim Fulton
On Fri, Dec 4, 2009 at 11:31 AM, Erik Dahl wrote: ... > I haven't dove into the relstorage yet but have heard it will perform > better. It doesn't. See: https://mail.zope.org/pipermail/zodb-dev/2009-October/012758.html -- Jim Fulton ___ For more i

Re: [ZODB-Dev] Making ZODB / ZEO faster

2009-12-04 Thread Erik Dahl
Right I have looked at the nosql stuff a good bit. MongoDB is interesting but no transactions and their sharding is alpha code. ChouchDB has transactions but no sharding at all. HBase has what I want but I can't get my brain around how to use the wacky schema. Unfortunately, this isn't a

Re: [ZODB-Dev] Making ZODB / ZEO faster

2009-12-04 Thread Andreas Jung
The project - at least enterprise-level projects - requires are careful choice of the tools and backends. The ZODB is the golden bullet for all and everything. Depending on the data model and the project needs you have to look at relational database or NOSQL databases as alternatives. And as you w

Re: [ZODB-Dev] Making ZODB / ZEO faster

2009-12-04 Thread Fred Drake
On Fri, Dec 4, 2009 at 9:41 AM, Erik Dahl wrote: >  I'm guessing the indexes > are a hotspot (haven't tested this out much though I guess b-tree's > buckets should alleviate this problem some).  (is there a persistent > queue around?) Cataloging certainly can be a hot spot. Check out zc.catalogq

[ZODB-Dev] Making ZODB / ZEO faster

2009-12-04 Thread Erik Dahl
Guys, We have a product written in python using ZODB/ZEO and I would like to improve the speed of database in general. Things that I have seen that I would like to improve some I understand and some not. 1. Loading of largish (but not too large object had a list with around 20K references