Re: [HACKERS] Bloom Filter lookup for hash joins

2013-07-23 Thread Atri Sharma
On Tue, Jul 23, 2013 at 7:32 PM, Greg Stark wrote: > This exact idea was discussed a whole back. I think it was even implemented. > > The problem Tom raised at the time is that the memory usage of the bloom > filter implies smaller or less efficient hash table. It's difficult to > determine whethe

Re: [HACKERS] Bloom Filter lookup for hash joins

2013-07-23 Thread Greg Stark
This exact idea was discussed a whole back. I think it was even implemented. The problem Tom raised at the time is that the memory usage of the bloom filter implies smaller or less efficient hash table. It's difficult to determine whether you're coming out ahead or behind. I think it should be po

Re: [HACKERS] Bloom Filter lookup for hash joins

2013-06-26 Thread Atri Sharma
On Thu, Jun 27, 2013 at 12:01 AM, Jeff Janes wrote: > I don't think that sounds all that promising. When the hash table does not > fit in memory, it is either partitioned into multiple passes, each of which > do fit in memory, or it chooses a different plan altogether. Yeah, my point is, we cou

Re: [HACKERS] Bloom Filter lookup for hash joins

2013-06-26 Thread Jeff Janes
On Wed, Jun 26, 2013 at 5:01 AM, Atri Sharma wrote: > > > The problem here is that if the hash table is in memory, doing a hash > > table lookup directly is likely to be cheaper than a bloom filter > > lookup, even if the bloom filter fits into the processor cache and the > > hash table doesn't (

Re: [HACKERS] Bloom Filter lookup for hash joins

2013-06-26 Thread Tom Lane
Ants Aasma writes: > On Wed, Jun 26, 2013 at 9:46 AM, Atri Sharma wrote: >> I have been reading the current implementation of hash joins, and in >> ExecScanHashBucket, which I understand is the actual lookup function, >> we could potentially look at a bloom filter per bucket. Instead of >> actual

Re: [HACKERS] Bloom Filter lookup for hash joins

2013-06-26 Thread Atri Sharma
On Wed, Jun 26, 2013 at 5:47 PM, Simon Riggs wrote: > On 26 June 2013 07:46, Atri Sharma wrote: > >> >> I have been researching bloom filters and discussed it on IRC with >> RhodiumToad and David Fetter, and they pointed me to the various >> places that could potentially have bloom filters, apart

Re: [HACKERS] Bloom Filter lookup for hash joins

2013-06-26 Thread Simon Riggs
On 26 June 2013 07:46, Atri Sharma wrote: > I have been researching bloom filters and discussed it on IRC with > RhodiumToad and David Fetter, and they pointed me to the various > places that could potentially have bloom filters, apart from the > places that already have them currently. > > I ha

Re: [HACKERS] Bloom Filter lookup for hash joins

2013-06-26 Thread Atri Sharma
> One idea that I had was to use them to do CLOG lookups from smaller > datastructures. You could store the list of aborted xids in bloom > filters. When a xid is not found in the filter, then it is known to > have committed, if the filter returns true, then you have to check the > real CLOG. This

Re: [HACKERS] Bloom Filter lookup for hash joins

2013-06-26 Thread Ants Aasma
On Wed, Jun 26, 2013 at 9:46 AM, Atri Sharma wrote: > I have been researching bloom filters and discussed it on IRC with > RhodiumToad and David Fetter, and they pointed me to the various > places that could potentially have bloom filters, apart from the > places that already have them currently.

Re: [HACKERS] Bloom Filter lookup for hash joins

2013-06-26 Thread Atri Sharma
On Wed, Jun 26, 2013 at 2:09 PM, Pavel Stehule wrote: > Hello > > it looks interesting - please, try to send some performance results, > > Regards > > Pavel Hi, Sure. I will think more about it and put up a design on the list soon. My current aim would be to work on hash joins. If it works well

Re: [HACKERS] Bloom Filter lookup for hash joins

2013-06-26 Thread Pavel Stehule
Hello it looks interesting - please, try to send some performance results, Regards Pavel 2013/6/26 Atri Sharma : > Hi All, > > I have been researching bloom filters and discussed it on IRC with > RhodiumToad and David Fetter, and they pointed me to the various > places that could potentially ha

[HACKERS] Bloom Filter lookup for hash joins

2013-06-25 Thread Atri Sharma
Hi All, I have been researching bloom filters and discussed it on IRC with RhodiumToad and David Fetter, and they pointed me to the various places that could potentially have bloom filters, apart from the places that already have them currently. I have been reading the current implementation of h