Re: multiview using bloom filters

2010-09-26 Thread Norman Barker
I have added the formatting changes and contacted the author of scalable bloom filters, it seems that bitarray (and the hipe version) came from a discussion on the erlang mailing lists http://groups.google.com/group/erlang-programming/browse_thread/thread/7c0191b1d709a5fe/ea5cf52b46d67d76?lnk=gst&

Re: multiview using bloom filters

2010-09-25 Thread Paul Davis
Although, I don't believe theirs is growable. But if it is, that might be interesting to test for speed. Or we could add the growable parts. On Sat, Sep 25, 2010 at 5:44 AM, Robert Dionne wrote: > Norman, > >   Basho also has a bloom filter implementation packaged as a separate > project[1], tha

Re: multiview using bloom filters

2010-09-25 Thread Robert Dionne
Norman, Basho also has a bloom filter implementation packaged as a separate project[1], that you might find useful. It's used in Bitcask. Cheers, Bob [1] http://github.com/basho/ebloom On Sep 24, 2010, at 11:21 PM, Norman Barker wrote: > Paul, > > yes, performance is actually much b

Re: multiview using bloom filters

2010-09-25 Thread Filipe David Manana
Hi, good work. I haven't had the time to look at it in detail, only glanced at it. My comments: - use indentation of 4 spaces - I see some places using indentation of 2 spaces, others using more, etc; - most of CouchDB's code uses a different style of case expression indentation - each case branc

Re: multiview using bloom filters

2010-09-24 Thread Paul Davis
I would say if you find it somewhere with an EPL header, then that's a good sign. But best to check with the original author that it was his intent. As an interesting aside, I'm not even sure if its possible that someone that's not an employee of Ericson can release something under the EPL. I'm no

Re: multiview using bloom filters

2010-09-24 Thread Norman Barker
Paul, yes, performance is actually much better (for some of our harder queries, so all docs over time with field X (two views), 10x faster), I am testing with docs that in total emit ~100K of keys (following the raindrop megaview). Some of the scalable bloom filter project contained EPL headers,

Re: multiview using bloom filters

2010-09-24 Thread Paul Davis
Norman, Just glanced through. Looks better. Any feeling for a performance differences? Also, I glanced at the original files that you linked to. The bit array files didn't have a license, but what you've got there does have EPL headers. We need to make sure we have permission to do so. I would as

multiview using bloom filters

2010-09-24 Thread Norman Barker
Hi, thanks to Paul's excellent suggestion I have rewritten the multiview to use bloom filters, I had a concern that a bloom filter per view would use too much memory but thanks in the main to excellent implementation of bloom filters in erlang (http://sites.google.com/site/scalablebloomfilters/) t