Re: [HACKERS] Bitmap index scans use of filters on available columns

2015-11-08 Thread Jeff Janes
On Sun, Nov 8, 2015 at 12:34 PM, Tom Lane wrote: > Jeff Janes writes: >> On Fri, Nov 6, 2015 at 7:15 PM, Tomas Vondra >> wrote: >>> I've however also noticed that all the 'like' procedures are marked as not >>> leak proof, which is a bit unfortunate because that's the example from >>> Jeff's e-m

Re: [HACKERS] Bitmap index scans use of filters on available columns

2015-11-08 Thread Tom Lane
Jeff Janes writes: > On Fri, Nov 6, 2015 at 7:15 PM, Tomas Vondra > wrote: >> I've however also noticed that all the 'like' procedures are marked as not >> leak proof, which is a bit unfortunate because that's the example from >> Jeff's e-mail that started this thread. > Is there a reason they a

Re: [HACKERS] Bitmap index scans use of filters on available columns

2015-11-08 Thread Jeff Janes
On Fri, Nov 6, 2015 at 7:15 PM, Tomas Vondra wrote: > Hi, > > On 11/07/2015 02:18 AM, Robert Haas wrote: >> >> On Fri, Nov 6, 2015 at 7:11 PM, Tomas Vondra >> wrote: I think LEAKPROOF is probably fine for this. How would the new thing be different? >>> >>> >>> I don't think so - A

Re: [HACKERS] Bitmap index scans use of filters on available columns

2015-11-06 Thread Amit Kapila
On Fri, Nov 6, 2015 at 10:28 AM, Kyotaro HORIGUCHI < horiguchi.kyot...@lab.ntt.co.jp> wrote: > > Hello, > > At Fri, 6 Nov 2015 09:49:30 +0530, Amit Kapila wrote in > > Apart from other problems discussed, I think it could also lead to > > a performance penality for the cases when the qual conditi

Re: [HACKERS] Bitmap index scans use of filters on available columns

2015-11-06 Thread Tomas Vondra
Hi, On 11/07/2015 02:18 AM, Robert Haas wrote: On Fri, Nov 6, 2015 at 7:11 PM, Tomas Vondra wrote: I think LEAKPROOF is probably fine for this. How would the new thing be different? I don't think so - AFAIK "leakproof" is about not revealing information about arguments, nothing more and not

Re: [HACKERS] Bitmap index scans use of filters on available columns

2015-11-06 Thread Robert Haas
On Fri, Nov 6, 2015 at 7:11 PM, Tomas Vondra wrote: >> I think LEAKPROOF is probably fine for this. How would the new thing >> be different? > > I don't think so - AFAIK "leakproof" is about not revealing information > about arguments, nothing more and nothing less. It does not say whether it's >

Re: [HACKERS] Bitmap index scans use of filters on available columns

2015-11-06 Thread Tomas Vondra
Hi, On 11/05/2015 07:36 PM, Robert Haas wrote: On Thu, Nov 5, 2015 at 1:29 PM, Tomas Vondra wrote: But then again, can we come up with a way to distinguish operators that are safe to evaluate on indexes - either automatic or manual? We already do that with the indexable operators with explici

Re: [HACKERS] Bitmap index scans use of filters on available columns

2015-11-05 Thread Kyotaro HORIGUCHI
Hello, At Fri, 6 Nov 2015 09:49:30 +0530, Amit Kapila wrote in > Apart from other problems discussed, I think it could also lead to > a performance penality for the cases when the qual condition is > costly as evaluating such a qual against lot of dead rows would be a > time consuming operation

Re: [HACKERS] Bitmap index scans use of filters on available columns

2015-11-05 Thread Amit Kapila
On Thu, Nov 5, 2015 at 7:45 AM, Tomas Vondra wrote: > Hi, > > On 11/04/2015 11:32 PM, Tom Lane wrote: > >> Jeff Janes writes: >> >>> On Wed, Nov 4, 2015 at 7:14 AM, Tom Lane wrote: >>> You're missing my point: that is possible in an indexscan, but *not* in a bitmap indexscan, because

Re: [HACKERS] Bitmap index scans use of filters on available columns

2015-11-05 Thread Robert Haas
On Thu, Nov 5, 2015 at 1:29 PM, Tomas Vondra wrote: >> Well, one thing is that you might leak information about >> already-deleted rows, which could be a security vulnerability, or more >> mundanely cause a function to error out when there are no actually >> visible rows that could trigger such an

Re: [HACKERS] Bitmap index scans use of filters on available columns

2015-11-05 Thread Tomas Vondra
Hi, On 11/05/2015 06:51 PM, Robert Haas wrote: On Wed, Nov 4, 2015 at 9:15 PM, Tomas Vondra wrote: I certainly understand there are cases that require care - like the leakproof thing pointed out by Robert for example. I don't immediately see why evaluation against dead rows would be a problem.

Re: [HACKERS] Bitmap index scans use of filters on available columns

2015-11-05 Thread Robert Haas
On Wed, Nov 4, 2015 at 9:15 PM, Tomas Vondra wrote: > I certainly understand there are cases that require care - like the > leakproof thing pointed out by Robert for example. I don't immediately see > why evaluation against dead rows would be a problem. Well, one thing is that you might leak info

Re: [HACKERS] Bitmap index scans use of filters on available columns

2015-11-04 Thread Tomas Vondra
Hi, On 11/04/2015 11:32 PM, Tom Lane wrote: Jeff Janes writes: On Wed, Nov 4, 2015 at 7:14 AM, Tom Lane wrote: You're missing my point: that is possible in an indexscan, but *not* in a bitmap indexscan, because the index AM APIs are totally different in the two cases. In a bitmap scan, nothi

Re: [HACKERS] Bitmap index scans use of filters on available columns

2015-11-04 Thread Tom Lane
Jeff Janes writes: > On Wed, Nov 4, 2015 at 7:14 AM, Tom Lane wrote: >> You're missing my point: that is possible in an indexscan, but *not* in a >> bitmap indexscan, because the index AM APIs are totally different in the >> two cases. In a bitmap scan, nothing more than a TID bitmap is ever >>

Re: [HACKERS] Bitmap index scans use of filters on available columns

2015-11-04 Thread Jeff Janes
On Wed, Nov 4, 2015 at 7:14 AM, Tom Lane wrote: > Simon Riggs writes: >> On 4 November 2015 at 15:54, Tom Lane wrote: >> We generate this plan >> Index Scan using f_x_y_idx on f (cost=0.42..26075.71 rows=209 width=37) >>Index Cond: (x = 5) >>Filter: (y ~~ '%abc%'::text) > >> So it shou

Re: [HACKERS] Bitmap index scans use of filters on available columns

2015-11-04 Thread Robert Haas
On Wed, Nov 4, 2015 at 10:59 AM, Tom Lane wrote: > Simon Riggs writes: >> On 4 November 2015 at 16:14, Tom Lane wrote: >>> You're missing my point: that is possible in an indexscan, but *not* in a >>> bitmap indexscan, because the index AM APIs are totally different in the >>> two cases. In a b

Re: [HACKERS] Bitmap index scans use of filters on available columns

2015-11-04 Thread Simon Riggs
On 4 November 2015 at 16:59, Tom Lane wrote: > Simon Riggs writes: > > On 4 November 2015 at 16:14, Tom Lane wrote: > >> You're missing my point: that is possible in an indexscan, but *not* in > a > >> bitmap indexscan, because the index AM APIs are totally different in the > >> two cases. In

Re: [HACKERS] Bitmap index scans use of filters on available columns

2015-11-04 Thread Tom Lane
Simon Riggs writes: > On 4 November 2015 at 16:14, Tom Lane wrote: >> You're missing my point: that is possible in an indexscan, but *not* in a >> bitmap indexscan, because the index AM APIs are totally different in the >> two cases. In a bitmap scan, nothing more than a TID bitmap is ever >> re

Re: [HACKERS] Bitmap index scans use of filters on available columns

2015-11-04 Thread Simon Riggs
On 4 November 2015 at 16:14, Tom Lane wrote: > Simon Riggs writes: > > On 4 November 2015 at 15:54, Tom Lane wrote: > > We generate this plan > > Index Scan using f_x_y_idx on f (cost=0.42..26075.71 rows=209 width=37) > >Index Cond: (x = 5) > >Filter: (y ~~ '%abc%'::text) > > > So it

Re: [HACKERS] Bitmap index scans use of filters on available columns

2015-11-04 Thread Tom Lane
Simon Riggs writes: > On 4 November 2015 at 15:54, Tom Lane wrote: > We generate this plan > Index Scan using f_x_y_idx on f (cost=0.42..26075.71 rows=209 width=37) >Index Cond: (x = 5) >Filter: (y ~~ '%abc%'::text) > So it should be possible to do the Filter condition on the BitmapInd

Re: [HACKERS] Bitmap index scans use of filters on available columns

2015-11-04 Thread Simon Riggs
On 4 November 2015 at 15:54, Tom Lane wrote: > Nicolas Barbier writes: > > 2015-11-04 Antonin Houska : > >> (see expand_indexqual_opclause()), I'm not sure any kind of expansion is > >> possible for '%abc%' which would result in a b-tree searchable > condition. > > > I think the question is not

Re: [HACKERS] Bitmap index scans use of filters on available columns

2015-11-04 Thread Tom Lane
Nicolas Barbier writes: > 2015-11-04 Antonin Houska : >> (see expand_indexqual_opclause()), I'm not sure any kind of expansion is >> possible for '%abc%' which would result in a b-tree searchable condition. > I think the question is not about using the b-tree for checking the > condition, but abo

Re: [HACKERS] Bitmap index scans use of filters on available columns

2015-11-04 Thread Nicolas Barbier
2015-11-04 Antonin Houska : > While prefix expression > > y like 'abc%' > > can be converted to > > y >= 'abc' > > (see expand_indexqual_opclause()), I'm not sure any kind of expansion is > possible for '%abc%' which would result in a b-tree searchable condition. I think the question is not about

Re: [HACKERS] Bitmap index scans use of filters on available columns

2015-11-04 Thread Antonin Houska
While prefix expression y like 'abc%' can be converted to y >= 'abc' (see expand_indexqual_opclause()), I'm not sure any kind of expansion is possible for '%abc%' which would result in a b-tree searchable condition. Jeff Janes wrote: > Is there a fundamental reason the filter on y is not bei

[HACKERS] Bitmap index scans use of filters on available columns

2015-11-03 Thread Jeff Janes
create table f as select (random()*100)::int as x, md5(random()::text) as y from generate_series(1,100); create index on f (x, y); analyze verbose f; --dont vacuum explain select * from f where x=5 and y like '%abc%'; QUERY PLAN -