I have done it, not with 1,000,000 docs but using megaviews that emitted lots and lots of keys.
Testing document id inclusion in a view and finding the size of a view are the bottle necks, the multiview streams the results so there isn't a great impact on the server and it was responsive enough for my needs. Norman On Fri, Sep 17, 2010 at 11:30 AM, Stephen Prater <[email protected]> wrote: > I'd say you're about twelve steps ahead of this then. It was pretty basic. > > I could shorter it - it was basically: > > You'll find the need to do these things, and you'll do them the way that > seems most right, but somebody else has probably already found a better way > of doing them, so don't reinvent the wheel. We've already got wheels. > > That said, sort-merge is a pretty good solution here I think. Did the > 1,000,000 doc tests ever get done? > > stephen > > On Sep 17, 2010, at 11:54 AM, Norman Barker wrote: > >> Stephen, >> >> if you could that would be greatly appreciated, I did read a lot of >> database theory before writing the multiview and settled on ordering >> the views from smallest to largest before doing the intersection in a >> multi-process ring (which now seems obvious!), but I am not an expert >> like the people from Postgres and I am sure there are smarter ways. >> >> thanks, >> >> Norman >> >> On Fri, Sep 17, 2010 at 10:47 AM, Stephen Prater <[email protected]> >> wrote: >>> >>> No, not specifically, but Josh Berkus from Postgres gave a good talk in >>> which he talked about JOINS (which multiview is basically a kind of) and >>> gave some brief suggestions for them relevant to your (several weeks >>> prior) >>> question about whether there was a more efficient strategy for finding >>> the >>> intersection of two views rather than iterating them. >>> >>> I'll see if I can dig up the slides or something. >>> >>> stephen >>> >>> On Sep 17, 2010, at 11:33 AM, Norman Barker wrote: >>> >>>> I wasn't at CouchCamp as I had a conflicting schedule, was the >>>> multiview discussed? >>>> >>>> On Fri, Sep 17, 2010 at 10:20 AM, Stephen Prater >>>> <[email protected]> >>>> wrote: >>>>> >>>>> Were you at CouchCamp, Norman? >>>>> >>>>> On Sep 17, 2010, at 9:25 AM, Norman Barker wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> the multiview is at >>>>>> >>>>>> http://github.com/normanb/couchdb >>>>>> >>>>>> I have included multiview tests in the test suite to show how it >>>>>> works. >>>>>> >>>>>> I would really like to get this into trunk so it can be improved and >>>>>> used! >>>>>> >>>>>> thanks, >>>>>> >>>>>> Norman >>>>>> >>>>>>> On Fri, Sep 17, 2010 at 7:37 AM, cdr53x <[email protected]> wrote: >>>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> I was wonderning if you could point me on some docs/examples on how >>>>>>>> the >>>>>>>> mutli view works ? >>>>>>>> >>>>>>>> Is this feature already in a packaged version and if not how should >>>>>>>> I >>>>>>>> proceed to benefit from this patch ? >>>>>>>> >>>>>>>> Thanks in advance, >>>>>>>> >>>>>>>> cdrx >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>> >>> >>> >> > >
