Re: [PERFORM] Query optimization using order by and limit

2011-09-25 Thread Tom Lane
Stephen Frost sfr...@snowman.net writes: What I think is happening here is that PG is pushing down that filter (not typically a bad thing..), but with that condition, it's going to scan the index until it finds a match for that filter before returning back up only to have that result cut out

Re: [PERFORM] Query optimization using order by and limit

2011-09-25 Thread Stephen Frost
* Tom Lane (t...@sss.pgh.pa.us) wrote: Yeah, it's spending quite a lot of time finding the first matching row in each child table. I'm curious why that is though; are the child tables not set up with nonoverlapping firstloadtime ranges? They are set up w/ nonoverlapping firstloadtime ranges,

Re: [PERFORM] Query optimization using order by and limit

2011-09-25 Thread Tom Lane
Stephen Frost sfr...@snowman.net writes: * Tom Lane (t...@sss.pgh.pa.us) wrote: Yeah, it's spending quite a lot of time finding the first matching row in each child table. I'm curious why that is though; are the child tables not set up with nonoverlapping firstloadtime ranges? The issue

Re: [PERFORM] Query optimization using order by and limit

2011-09-22 Thread k...@rice.edu
On Wed, Sep 21, 2011 at 11:22:53PM -0400, Tom Lane wrote: Michael Viscuso michael.visc...@getcarbonblack.com writes: Greg/Tom, you are correct, these columns should be modified to whatever is easiest for Postgres to recognize 64-bit unsigned integers. Would you still recommend bigint for

Re: [PERFORM] Query optimization using order by and limit

2011-09-22 Thread Michael Viscuso
Thanks Ken, I'm discussing with my coworker how to best make that change *as we speak*. Do you think this will also resolve the original issue I'm seeing where the query doesn't limit out properly and spends time in child tables that won't yield any results? I was hoping that by using the check

Re: [PERFORM] Query optimization using order by and limit

2011-09-22 Thread Stephen Frost
* Michael Viscuso (michael.visc...@getcarbonblack.com) wrote: Adding the final condition hosts_guid = '2007075705813916178' is what ultimately kills it http://explain.depesz.com/s/8zy. By adding the host_guid, it spends considerably more time in the older tables than without this condition

Re: [PERFORM] Query optimization using order by and limit

2011-09-22 Thread Michael Viscuso
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Stephen, I spent the better part of the day implementing an application layer nested loop and it seems to be working well. Of course it's a little slower than a Postgres only solution because it has to pass data back and forth for each daily

Re: [PERFORM] Query optimization using order by and limit

2011-09-22 Thread Stephen Frost
Mike, * Michael Viscuso (michael.visc...@getcarbonblack.com) wrote: I spent the better part of the day implementing an application layer nested loop and it seems to be working well. Of course it's a little slower than a Postgres only solution because it has to pass data back and forth for

Re: [PERFORM] Query optimization using order by and limit

2011-09-22 Thread Michael Viscuso
Stephen, Yes, I couldn't agree more. The next two things I will be looking at very carefully are the timestamps and indexes. I will reply to this post if either dramatically helps. Thanks again for all your help. My eyes were starting to bleed from staring at explain logs! Mike On Thu, Sep

[PERFORM] Query optimization using order by and limit

2011-09-21 Thread Michael Viscuso
First of all, thank you for taking the time to review my question. After attending the PostgresOpen conference in Chicago last week, I've been pouring over explain logs for hours on end and although my system is MUCH better, I still can't resolve a few issues. Luckily my data is pretty well

Re: [PERFORM] Query optimization using order by and limit

2011-09-21 Thread Greg Smith
On 09/21/2011 07:14 PM, Michael Viscuso wrote: Check constraints: osmoduleloads_2011_09_14_event_time_check CHECK (event_time = '2011-09-14 00:00:00'::timestamp without time zone) osmoduleloads_2011_09_14_firstloadtime_check CHECK (firstloadtime = 1296044640::bigint::numeric

Re: [PERFORM] Query optimization using order by and limit

2011-09-21 Thread Tom Lane
Greg Smith g...@2ndquadrant.com writes: That weird casting can't be helping. I'm not sure if it's your problem here, but the constraint exclusion code is pretty picky about matching the thing you're looking for against the CHECK constraint, and this is a messy one. The bigint conversion

Re: [PERFORM] Query optimization using order by and limit

2011-09-21 Thread Michael Viscuso
Thanks guys, First of all, I should have included my postgres.conf file with the original submission. Sorry about that. It is now attached. Based on a recommendation, I also should have shown the parent child relationship between osmoduleloads and its daily partitioned tables. to reduce

Re: [PERFORM] Query optimization using order by and limit

2011-09-21 Thread Tom Lane
Michael Viscuso michael.visc...@getcarbonblack.com writes: Greg/Tom, you are correct, these columns should be modified to whatever is easiest for Postgres to recognize 64-bit unsigned integers. Would you still recommend bigint for unsigned integers? I likely read the wrong documentation that