Re: Partitioned Clusters

2009-02-21 Thread Ben Browning
On Fri, Feb 20, 2009 at 7:34 PM, Chris Anderson wrote: > I think so. I think that there could be proxy overlap / redundancy > across all levels of the tree, and also in the case of a flat tree. > > As long as the proxies agree on how to hash from URLs to nodes it > should just work. I've been thi

Re: Partitioned Clusters

2009-02-20 Thread Chris Anderson
On Fri, Feb 20, 2009 at 4:15 PM, Mike Malone wrote: > Hi, I don't think I've commented on this list before so let me briefly > introduce myself. I'm Mike Malone. I live in San Francisco. I'm a developer > (primarily web dev) and have some experience working with large clustered > databases. I work

Re: Partitioned Clusters

2009-02-20 Thread Mike Malone
Hi, I don't think I've commented on this list before so let me briefly introduce myself. I'm Mike Malone. I live in San Francisco. I'm a developer (primarily web dev) and have some experience working with large clustered databases. I worked for Pownce.com, but moved to Six Apart when they acquired

Re: Partitioned Clusters

2009-02-20 Thread Chris Anderson
On Fri, Feb 20, 2009 at 2:45 PM, Damien Katz wrote: > > On Feb 20, 2009, at 4:37 PM, Stefan Karpinski wrote: > >>> >>> Trees would be overkill except for with very large clusters. >>> >> >>> With CouchDB map views, you need to combine results from every node in a >>> big merge sort. If you combine

Re: Partitioned Clusters

2009-02-20 Thread Chris Anderson
On Fri, Feb 20, 2009 at 1:37 PM, Stefan Karpinski wrote: > > That makes sense and it clarifies one of my questions about this topic. Is > the goal of partitioned clustering to increase performance for very large > data sets, or to increase reliability? It would seem from this answere that > the g

Re: Partitioned Clusters

2009-02-20 Thread Damien Katz
On Feb 20, 2009, at 4:37 PM, Stefan Karpinski wrote: Trees would be overkill except for with very large clusters. With CouchDB map views, you need to combine results from every node in a big merge sort. If you combine all results at a single node, the single clients ability to simultane

Re: Partitioned Clusters

2009-02-20 Thread Stefan Karpinski
> > Trees would be overkill except for with very large clusters. > > With CouchDB map views, you need to combine results from every node in a > big merge sort. If you combine all results at a single node, the single > clients ability to simultaneously pull data and sort data from all other > nodes

Re: Partitioned Clusters

2009-02-20 Thread Damien Katz
On Feb 20, 2009, at 1:55 PM, Stefan Karpinski wrote: Hi, I thought I'd introduce myself since I'm new here on the couchdb list. I'm Stefan Karpinski. I've worked in the Monitoring Group at Akamai, Operations R&D at Citrix Online, and I'm nearly done with a PhD in computer networking at the mome

Re: Partitioned Clusters

2009-02-20 Thread Chris Anderson
On Fri, Feb 20, 2009 at 10:55 AM, Stefan Karpinski wrote: > Hi, I thought I'd introduce myself since I'm new here on the couchdb > list. I'm Stefan Karpinski. I've worked in the Monitoring Group at > Akamai, Operations R&D at Citrix Online, and I'm nearly done with a > PhD in computer networking a

Re: Partitioned Clusters

2009-02-20 Thread Stefan Karpinski
Hi, I thought I'd introduce myself since I'm new here on the couchdb list. I'm Stefan Karpinski. I've worked in the Monitoring Group at Akamai, Operations R&D at Citrix Online, and I'm nearly done with a PhD in computer networking at the moment. So I guess I've thought about this kind of stuff a bi

Re: Partitioned Clusters

2009-02-20 Thread Robert Newson
Any thoughts as to how (or even if) this tree-wise result aggregation would work for externals? I'm thinking specifically about couchdb-lucene, where multi-node results aggregation is possible, given a framework like you propose here. The results that couchdb-lucene produces can already be aggrega

Re: Partitioned Clusters

2009-02-19 Thread Chris Anderson
On Thu, Feb 19, 2009 at 6:39 PM, Ben Browning wrote: > Overall the model sounds very similar to what I was thinking. I just > have a few comments. > >> In this model documents are saved to a leaf node depending on a hash >> of the docid. This means that lookups are easy, and need only to touch >>

Re: Partitioned Clusters

2009-02-19 Thread Ben Browning
Overall the model sounds very similar to what I was thinking. I just have a few comments. > In this model documents are saved to a leaf node depending on a hash > of the docid. This means that lookups are easy, and need only to touch > the leaf node which holds the doc. Redundancy can be provided