Re: [PERFORM] Gist indexing performance with cidr types

2015-08-28 Thread Henrik Thostrup Jensen
Hi On Thu, 27 Aug 2015, Emre Hasegeli wrote: I think the slowdown is not related with the key your searched for, but the organisation of the index. We have a simple structure for the index keys. Basically, common bits of the child nodes are stored on the parent node. It leads to not efficien

Re: [PERFORM] Gist indexing performance with cidr types

2015-08-27 Thread Emre Hasegeli
> Nothing really interesting here though. I think the slowdown is not related with the key your searched for, but the organisation of the index. We have a simple structure for the index keys. Basically, common bits of the child nodes are stored on the parent node. It leads to not efficient inde

Re: [PERFORM] Gist indexing performance with cidr types

2015-08-27 Thread Henrik Thostrup Jensen
On Wed, 26 Aug 2015, Jeff Janes wrote: Any chance you can share the actual underlying data? Sure. I added a snapshot to the repo: https://github.com/job/irrexplorer/blob/master/data/irrexplorer_dump.sql.gz?raw=true I noticed it wasn't on github, but is that because it is proprietary, or just

Re: [PERFORM] Gist indexing performance with cidr types

2015-08-27 Thread Henrik Thostrup Jensen
On Wed, 26 Aug 2015, Emre Hasegeli wrote: Can you try to isolate it even more by something like this: I tried some different bisection approaches: -- base query (time ~19 seconds) EXPLAIN (ANALYZE, BUFFERS) SELECT rv.route, rv.asn, rv.source FROM (SELECT DISTINCT route FROM routes_view WH

Re: [PERFORM] Gist indexing performance with cidr types

2015-08-26 Thread Jeff Janes
On Wed, Aug 26, 2015 at 4:29 AM, Henrik Thostrup Jensen wrote: > On Wed, 26 Aug 2015, Emre Hasegeli wrote: > > Are the coverage operatons just that expensive? >>> >> >> They shouldn't be. A similar query like yours works in 0.5 second on my >> laptop: >> > [snip] > > I get the same from your tes

Re: [PERFORM] Gist indexing performance with cidr types

2015-08-26 Thread Emre Hasegeli
> Then there is the mixed IPv6 and IPv4 data that might factor in. It shouldn't be the problem. The index should separate them on the top level. > I tried the approach from your benchmark, to try make a more isolated test > case: Can you try to isolate it even more by something like this: sele

Re: [PERFORM] Gist indexing performance with cidr types

2015-08-26 Thread Henrik Thostrup Jensen
On Wed, 26 Aug 2015, Emre Hasegeli wrote: Are the coverage operatons just that expensive? They shouldn't be. A similar query like yours works in 0.5 second on my laptop: [snip] I get the same from your testcase. Maybe, something we haven't expected about your dataset causes a performance

Re: [PERFORM] Gist indexing performance with cidr types

2015-08-26 Thread Emre Hasegeli
> Are the coverage operatons just that expensive? They shouldn't be. A similar query like yours works in 0.5 second on my laptop: ># create table inner_side as select i, ((random() * 255.5)::int::text || '.' >|| (random() * 255.5)::int::text || '.' || (random() * 255.5)::int::text || >'.' || (

Re: [PERFORM] Gist indexing performance with cidr types

2015-08-26 Thread Henrik Thostrup Jensen
On Wed, 26 Aug 2015, Henrik Thostrup Jensen wrote: Can you try 9.5 to see if they help? I'll try installing it and report back. I upgraded to 9.5 (easier than expected) and ran vacuum analyze. The query planner now chooses index scan for outer and inner join. This seems to cut off roughly

Re: [PERFORM] Gist indexing performance with cidr types

2015-08-26 Thread Henrik Thostrup Jensen
Hi, thanks for the reply. On Tue, 25 Aug 2015, Emre Hasegeli wrote: I'm trying to get a query to run fast enough for interactive use. I've gotten some speed-up, but still not there. It is for a tool called IRRExplorer (http://irrexplorer.nlnog.net/) that correlates IP routes between Internet Ro

Re: [PERFORM] Gist indexing performance with cidr types

2015-08-25 Thread Emre Hasegeli
> I'm trying to get a query to run fast enough for interactive use. I've gotten > some speed-up, but still not there. It is for a tool called IRRExplorer > (http://irrexplorer.nlnog.net/) that correlates IP routes between Internet > Route Registries and real-world routing information. We landed on

[PERFORM] Gist indexing performance with cidr types

2015-08-25 Thread Henrik Thostrup Jensen
Hi I'm trying to get a query to run fast enough for interactive use. I've gotten some speed-up, but still not there. It is for a tool called IRRExplorer (http://irrexplorer.nlnog.net/) that correlates IP routes between Internet Route Registries and real-world routing information. We landed on Pos