Bob, I can and have been testing the multiview at this scale, it is ok (fast enough), but I think being able to test inclusion of a document id in a view without having to loop would be a considerable speed improvement. If you have any ideas let me know.
thanks, Norman On Mon, Aug 30, 2010 at 10:49 AM, Robert Newson <[email protected]> wrote: > I'm sorry, I've had no time to play with this at scale. > > On Mon, Aug 30, 2010 at 5:35 PM, Norman Barker <[email protected]> > wrote: >> Hi, >> >> are there any more comments on this, if not can you describe the >> process (in particular how to obtain a wiki and jira account for >> couchdb which I have been unable to do) and I will start documenting >> this so we can put this into the trunk. >> >> Bob, were you able to do any more testing with large views, are there >> any suggestions on how to speed up the document id inclusion test as >> described below? >> >> thanks, >> >> Norman >> >> On Mon, Aug 23, 2010 at 9:22 AM, Norman Barker <[email protected]> >> wrote: >>> Bob, >>> >>> thanks for the feedback and for taking a look at the code. Guidelines >>> on when to use a supervisor within couchdb with a gen_server would be >>> appreciated, currently I have a supervisor and a gen_server, but if >>> couchdb has a supervision process I could remove that layer. >>> >>> I think plugins is a great idea, however intersection of views is such >>> as common request, perhaps there needs to plugin system and if a >>> plugin is rated enough it goes into trunk as a core feature. >>> >>> the four (or slightly more) summary is here >>> >>> http://github.com/normanb/couchdb/raw/trunk/src/couchdb/couch_query_ring.erl >>> >>> % >>> % send an id from the start list to the next node in the ring, if the >>> id is in adjacent node then the this node sends to the next ring node >>> .... >>> % if the id gets all round the ring and back to the start node then is >>> has intersected all queries and should be included. The nodes in the >>> ring >>> % should be sorted in size from small to large for this to be effective >>> % >>> % In addition send the initial id list round in parallel >>> >>> it really needs some eyes from the core couchdb coders to see how to >>> speed up the inclusion testing, looping is bad even if it is done in >>> parallel. >>> >>> Multiview is usable, I am using it with some pretty big mega-views (as >>> per the raindrop) model, I am also available to add features to this >>> as this is core part of our work and we want to give it to couch as a >>> contribution. >>> >>> thanks, >>> >>> Norman >>> >>> On Mon, Aug 23, 2010 at 5:05 AM, Robert Dionne >>> <[email protected]> wrote: >>>> Hi Norman, >>>> >>>> I took a peek at multiview. I haven't followed this too closely on the >>>> mailing list but this is *view intersection*? Is there a 5 line summary of >>>> what this does somewhere? >>>> >>>> I'm curious as to why the daemon needs to be a supervisor, most if not >>>> all of the other daemons are gen_servers. OTP allows this but I think this >>>> is a good area where some CouchDB guidelines on plugins would apply. >>>> >>>> It strikes me that views, the use of map/reduce, etc. are one of the >>>> trickier aspects of using CouchDB, particularly for new users coming from >>>> the SQL world. People are also reporting issues with performance of views, >>>> I guess often because reduce functions go out of control. >>>> >>>> I think the project would be better served if features like this were >>>> available as plugins. I would put GeoCouch in the same category. Its very >>>> neat and timely (given everyone wants to know where everyone else is using >>>> their telephone but without talking other than asynchronously), but a >>>> server plugin architecture that would allow this to be done cleanly should >>>> come first. >>>> >>>> This is just my opinion. I'd love to see some of the project founders and >>>> committers weigh in on this and set some direction. >>>> >>>> Best regards, >>>> >>>> Bob >>>> >>>> >>>> >>>> >>>> >>>> On Aug 22, 2010, at 5:45 PM, Norman Barker wrote: >>>> >>>>> I would like to take this multiview code and have it added to trunk if >>>>> possible, what are the next steps? >>>>> >>>>> thanks, >>>>> >>>>> Norman >>>>> >>>>> On Wed, Aug 18, 2010 at 11:44 AM, Norman Barker <[email protected]> >>>>> wrote: >>>>>> I have made >>>>>> >>>>>> http://github.com/normanb/couchdb >>>>>> >>>>>> which is a fork of the latest couchdb trunk with the multiview code >>>>>> and tests added. >>>>>> >>>>>> If geocouch is available then it can still be used. >>>>>> >>>>>> There are a couple of questions about the multiview on the user /dev >>>>>> list so I will be adding some more test cases during today. >>>>>> >>>>>> thanks, >>>>>> >>>>>> Norman >>>>>> >>>>>> On Tue, Aug 17, 2010 at 9:23 PM, Norman Barker <[email protected]> >>>>>> wrote: >>>>>>> this is possible, I forked geocouch since I use it, but I have already >>>>>>> separated the geocouch dependencies from the trunk. >>>>>>> >>>>>>> I can do this tomorrow, certainly be interested in any feedback. >>>>>>> >>>>>>> thanks, >>>>>>> >>>>>>> Norman >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Tue, Aug 17, 2010 at 7:49 PM, Volker Mische >>>>>>> <[email protected]> wrote: >>>>>>>> On 08/18/2010 03:26 AM, J Chris Anderson wrote: >>>>>>>>> >>>>>>>>> On Aug 16, 2010, at 4:38 PM, Norman Barker wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I have made the changes as recommended, adding a test case >>>>>>>>>> multiview.js and also adding the userCtx to open the db. >>>>>>>>>> >>>>>>>>>> I have also forked geocouch and this is available here >>>>>>>>>> >>>>>>>>> >>>>>>>>> this patch seems important (especially as people are already asking >>>>>>>>> for >>>>>>>>> help using it on user@) >>>>>>>>> >>>>>>>>> to get it committed, it either must remove the dependency on >>>>>>>>> GeoCouch, or >>>>>>>>> become part of CouchDB when (and if) GeoCouch becomes part of CouchDB. >>>>>>>>> >>>>>>>>> Is it possible / useful to make a version that doesn't use GeoCouch? >>>>>>>>> And >>>>>>>>> then to make the GeoCouch capabilities part GeoCouch for now? >>>>>>>>> >>>>>>>>> Chris >>>>>>>>> >>>>>>>> >>>>>>>> Hi Norman, >>>>>>>> >>>>>>>> if the patch is ready for trunk, I'd be happy to move the GeoCouch >>>>>>>> bits to >>>>>>>> GeoCouch itself (as GeoCouch isn't ready for trunk yet). >>>>>>>> >>>>>>>> Lately I haven't been that responsive when it comes to GeoCouch, but >>>>>>>> that >>>>>>>> will change (in about a month) after holidays and FOSS4G. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Volker >>>>>>>> >>>>>>> >>>>>> >>>> >>>> >>> >> >
