Re: [PERFORM] experiments in query optimization

2010-04-01 Thread Faheem Mitha
On Thu, 1 Apr 2010, Eliot Gable wrote: On Thu, Apr 1, 2010 at 3:01 PM, Faheem Mitha wrote: So, should I add indexes on the individual foreign key cols idlink_id and anno_id after all? I doubt that would help. You're sure of this? It is

Re: [PERFORM] experiments in query optimization

2010-04-01 Thread Eliot Gable
On Thu, Apr 1, 2010 at 3:01 PM, Faheem Mitha wrote: > > So, should I add indexes on the individual foreign key cols idlink_id >>> and anno_id after all? >>> >> >> I doubt that would help. >> > > You're sure of this? > > > It is always best to test and be certain.

Re: [PERFORM] experiments in query optimization

2010-04-01 Thread Faheem Mitha
On Thu, 1 Apr 2010, Robert Haas wrote: On Thu, Apr 1, 2010 at 2:15 PM, Faheem Mitha wrote: I had set the foreign keys in question (on the geno table) to be primary keys. This is because this setup is basically a glorified spreadsheet, and I don't want more than one cell corresponding to a

Re: [PERFORM] experiments in query optimization

2010-04-01 Thread Robert Haas
On Thu, Apr 1, 2010 at 2:15 PM, Faheem Mitha wrote: > I had set the foreign keys in question (on the geno table) to be primary > keys. This is because this setup is basically a glorified spreadsheet, and I > don't want more than one cell corresponding to a particular tuple of > idlink.id and anno.

Re: [PERFORM] experiments in query optimization

2010-04-01 Thread Faheem Mitha
Hi Eliot, Thanks for the comment. On Thu, 1 Apr 2010, Eliot Gable wrote: On Thu, Apr 1, 2010 at 7:46 AM, Faheem Mitha wrote: Looking at this more closely, idlink_id and anno_id are primary keys, so already have indexes on them, so my understanding (from the docs) is there is no purpose i

Re: [PERFORM] experiments in query optimization

2010-04-01 Thread Eliot Gable
On Thu, Apr 1, 2010 at 7:46 AM, Faheem Mitha wrote: > Looking at this more closely, idlink_id and anno_id are primary keys, so > already have indexes on them, so my understanding (from the docs) is there > is no purpose in creating them. That's why I removed the indexes that were > there (back l

Re: [PERFORM] experiments in query optimization

2010-04-01 Thread Faheem Mitha
On Wed, 31 Mar 2010, Faheem Mitha wrote: On Tue, 30 Mar 2010, Robert Haas wrote: On Tue, Mar 30, 2010 at 12:30 PM, Faheem Mitha You might need to create some indices, too. Ok. To what purpose? This query picks up everything from the tables and the planner does table scans, so conventio

Re: [PERFORM] experiments in query optimization

2010-03-31 Thread Robert Haas
On Wed, Mar 31, 2010 at 6:10 AM, Faheem Mitha wrote: > > [If Kevin Grittner reads this, please fix your email address. I am getting > bounces from your email address.] > > On Tue, 30 Mar 2010, Robert Haas wrote: > >> On Tue, Mar 30, 2010 at 12:30 PM, Faheem Mitha >> wrote: >>> >>> Sure, but defin

Re: [PERFORM] experiments in query optimization

2010-03-31 Thread Faheem Mitha
[If Kevin Grittner reads this, please fix your email address. I am getting bounces from your email address.] On Tue, 30 Mar 2010, Robert Haas wrote: On Tue, Mar 30, 2010 at 12:30 PM, Faheem Mitha wrote: Sure, but define sane setting, please. I guess part of the point is that I'm trying to

Re: [PERFORM] experiments in query optimization

2010-03-31 Thread Faheem Mitha
On Wed, 31 Mar 2010, Matthew Wakeling wrote: On Tue, 30 Mar 2010, Faheem Mitha wrote: work_mem = 1 GB (see diag.{tex/pdf}). Sure, but define sane setting, please. I guess part of the point is that I'm trying to keep memory low You're trying to keep memory usage low, but you have work_mem

Re: [PERFORM] experiments in query optimization

2010-03-31 Thread Matthew Wakeling
On Tue, 30 Mar 2010, Faheem Mitha wrote: work_mem = 1 GB (see diag.{tex/pdf}). Sure, but define sane setting, please. I guess part of the point is that I'm trying to keep memory low You're trying to keep memory usage low, but you have work_mem set to 1GB? Matthew -- "Prove to thyself that

Re: [PERFORM] experiments in query optimization

2010-03-30 Thread Robert Haas
On Tue, Mar 30, 2010 at 12:30 PM, Faheem Mitha wrote: > Sure, but define sane setting, please. I guess part of the point is that I'm > trying to keep memory low, and it seems this is not part of the planner's > priorities. That it, it does not take memory usage into consideration when > choosing a

Re: [PERFORM] experiments in query optimization

2010-03-30 Thread Faheem Mitha
On thing which I haven't really mentioned in this thread or in my writeup, is that the planners value for the number of rows in geno is way off base some of the time. It is around 800 million, it thinks it is 100 million. I don't know if this is significant or not, or what to do about it. eg

Re: [PERFORM] experiments in query optimization

2010-03-30 Thread Faheem Mitha
On Tue, 30 Mar 2010, Kevin Grittner wrote: Faheem Mitha wrote: If you're concerned about memory usage, try reducing work_mem; you've probably got it set to something huge. work_mem = 1 GB (see diag.{tex/pdf}). The point isn't that I'm using so much memory. Again, my question is, why are

Re: [PERFORM] experiments in query optimization

2010-03-30 Thread Kevin Grittner
Faheem Mitha wrote: >> If you're concerned about memory usage, try reducing work_mem; >> you've probably got it set to something huge. > > work_mem = 1 GB (see diag.{tex/pdf}). > > The point isn't that I'm using so much memory. Again, my question > is, why are these changes affecting memory us

Re: [PERFORM] experiments in query optimization

2010-03-29 Thread Faheem Mitha
On Mon, 29 Mar 2010, Robert Haas wrote: On Mon, Mar 29, 2010 at 2:31 PM, Faheem Mitha wrote: It's not really too clear to me from reading this what specific questions you're trying to answer. Quote from opt.{tex/pdf}, Section 1: "If I have to I can use Section~\ref{ped_hybrid} and Section

Re: [PERFORM] experiments in query optimization

2010-03-29 Thread Robert Haas
On Mon, Mar 29, 2010 at 2:31 PM, Faheem Mitha wrote: >> It's not really too clear to me from reading this what specific >> questions you're trying to answer. > > Quote from opt.{tex/pdf}, Section 1: > > "If I have to I can use Section~\ref{ped_hybrid} and > Section~\ref{tped_hybrid}, but I am left

Re: [PERFORM] experiments in query optimization

2010-03-29 Thread Faheem Mitha
On Mon, 29 Mar 2010, Robert Haas wrote: On Thu, Mar 25, 2010 at 3:57 PM, Faheem Mitha wrote: Hi everyone, I've been trying to reduce both memory usage and runtime for a query. Comments/suggestions gratefully received. Details are at http://bulldog.duhs.duke.edu/~faheem/snppy/opt.pdf See

Re: [PERFORM] experiments in query optimization

2010-03-29 Thread Robert Haas
On Thu, Mar 25, 2010 at 3:57 PM, Faheem Mitha wrote: > > Hi everyone, > > I've been trying to reduce both memory usage and runtime for a query. > Comments/suggestions gratefully received. Details are at > > http://bulldog.duhs.duke.edu/~faheem/snppy/opt.pdf > > See particularly Section 1 - Backgro

[PERFORM] experiments in query optimization

2010-03-25 Thread Faheem Mitha
Hi everyone, I've been trying to reduce both memory usage and runtime for a query. Comments/suggestions gratefully received. Details are at http://bulldog.duhs.duke.edu/~faheem/snppy/opt.pdf See particularly Section 1 - Background and Discussion. If you want a text version, see http://bull