Re: [PERFORM] experiments in query optimization

2010-04-01 Thread Faheem Mitha
On Wed, 31 Mar 2010, Faheem Mitha wrote: On Tue, 30 Mar 2010, Robert Haas wrote: On Tue, Mar 30, 2010 at 12:30 PM, Faheem Mitha fah...@email.unc.edu You might need to create some indices, too. Ok. To what purpose? This query picks up everything from the tables and the planner does

Re: [PERFORM] experiments in query optimization

2010-04-01 Thread Eliot Gable
On Thu, Apr 1, 2010 at 7:46 AM, Faheem Mitha fah...@email.unc.edu wrote: Looking at this more closely, idlink_id and anno_id are primary keys, so already have indexes on them, so my understanding (from the docs) is there is no purpose in creating them. That's why I removed the indexes that

Re: [PERFORM] experiments in query optimization

2010-04-01 Thread Faheem Mitha
Hi Eliot, Thanks for the comment. On Thu, 1 Apr 2010, Eliot Gable wrote: On Thu, Apr 1, 2010 at 7:46 AM, Faheem Mitha fah...@email.unc.edu wrote: Looking at this more closely, idlink_id and anno_id are primary keys, so already have indexes on them, so my understanding (from the docs) is

Re: [PERFORM] experiments in query optimization

2010-04-01 Thread Robert Haas
On Thu, Apr 1, 2010 at 2:15 PM, Faheem Mitha fah...@email.unc.edu wrote: I had set the foreign keys in question (on the geno table) to be primary keys. This is because this setup is basically a glorified spreadsheet, and I don't want more than one cell corresponding to a particular tuple of

Re: [PERFORM] experiments in query optimization

2010-04-01 Thread Faheem Mitha
On Thu, 1 Apr 2010, Robert Haas wrote: On Thu, Apr 1, 2010 at 2:15 PM, Faheem Mitha fah...@email.unc.edu wrote: I had set the foreign keys in question (on the geno table) to be primary keys. This is because this setup is basically a glorified spreadsheet, and I don't want more than one

Re: [PERFORM] experiments in query optimization

2010-04-01 Thread Eliot Gable
On Thu, Apr 1, 2010 at 3:01 PM, Faheem Mitha fah...@email.unc.edu wrote: So, should I add indexes on the individual foreign key cols idlink_id and anno_id after all? I doubt that would help. You're sure of this? It is always best to test and be certain.

Re: [PERFORM] experiments in query optimization

2010-04-01 Thread Faheem Mitha
On Thu, 1 Apr 2010, Eliot Gable wrote: On Thu, Apr 1, 2010 at 3:01 PM, Faheem Mitha fah...@email.unc.edu wrote: So, should I add indexes on the individual foreign key cols idlink_id and anno_id after all? I doubt that would help. You're

Re: [PERFORM] experiments in query optimization

2010-03-31 Thread Matthew Wakeling
On Tue, 30 Mar 2010, Faheem Mitha wrote: work_mem = 1 GB (see diag.{tex/pdf}). Sure, but define sane setting, please. I guess part of the point is that I'm trying to keep memory low You're trying to keep memory usage low, but you have work_mem set to 1GB? Matthew -- Prove to thyself that

Re: [PERFORM] experiments in query optimization

2010-03-31 Thread Faheem Mitha
On Wed, 31 Mar 2010, Matthew Wakeling wrote: On Tue, 30 Mar 2010, Faheem Mitha wrote: work_mem = 1 GB (see diag.{tex/pdf}). Sure, but define sane setting, please. I guess part of the point is that I'm trying to keep memory low You're trying to keep memory usage low, but you have

Re: [PERFORM] experiments in query optimization

2010-03-31 Thread Robert Haas
On Wed, Mar 31, 2010 at 6:10 AM, Faheem Mitha fah...@email.unc.edu wrote: [If Kevin Grittner reads this, please fix your email address. I am getting bounces from your email address.] On Tue, 30 Mar 2010, Robert Haas wrote: On Tue, Mar 30, 2010 at 12:30 PM, Faheem Mitha fah...@email.unc.edu

Re: [PERFORM] experiments in query optimization

2010-03-30 Thread Kevin Grittner
Faheem Mitha fah...@email.unc.edu wrote: If you're concerned about memory usage, try reducing work_mem; you've probably got it set to something huge. work_mem = 1 GB (see diag.{tex/pdf}). The point isn't that I'm using so much memory. Again, my question is, why are these changes

Re: [PERFORM] experiments in query optimization

2010-03-30 Thread Faheem Mitha
On Tue, 30 Mar 2010, Kevin Grittner wrote: Faheem Mitha fah...@email.unc.edu wrote: If you're concerned about memory usage, try reducing work_mem; you've probably got it set to something huge. work_mem = 1 GB (see diag.{tex/pdf}). The point isn't that I'm using so much memory. Again, my

Re: [PERFORM] experiments in query optimization

2010-03-30 Thread Faheem Mitha
On thing which I haven't really mentioned in this thread or in my writeup, is that the planners value for the number of rows in geno is way off base some of the time. It is around 800 million, it thinks it is 100 million. I don't know if this is significant or not, or what to do about it.

Re: [PERFORM] experiments in query optimization

2010-03-30 Thread Robert Haas
On Tue, Mar 30, 2010 at 12:30 PM, Faheem Mitha fah...@email.unc.edu wrote: Sure, but define sane setting, please. I guess part of the point is that I'm trying to keep memory low, and it seems this is not part of the planner's priorities. That it, it does not take memory usage into consideration

Re: [PERFORM] experiments in query optimization

2010-03-29 Thread Robert Haas
On Thu, Mar 25, 2010 at 3:57 PM, Faheem Mitha fah...@email.unc.edu wrote: Hi everyone, I've been trying to reduce both memory usage and runtime for a query. Comments/suggestions gratefully received. Details are at http://bulldog.duhs.duke.edu/~faheem/snppy/opt.pdf See particularly Section

Re: [PERFORM] experiments in query optimization

2010-03-29 Thread Faheem Mitha
On Mon, 29 Mar 2010, Robert Haas wrote: On Thu, Mar 25, 2010 at 3:57 PM, Faheem Mitha fah...@email.unc.edu wrote: Hi everyone, I've been trying to reduce both memory usage and runtime for a query. Comments/suggestions gratefully received. Details are at

Re: [PERFORM] experiments in query optimization

2010-03-29 Thread Robert Haas
On Mon, Mar 29, 2010 at 2:31 PM, Faheem Mitha fah...@email.unc.edu wrote: It's not really too clear to me from reading this what specific questions you're trying to answer. Quote from opt.{tex/pdf}, Section 1: If I have to I can use Section~\ref{ped_hybrid} and Section~\ref{tped_hybrid},

Re: [PERFORM] experiments in query optimization

2010-03-29 Thread Faheem Mitha
On Mon, 29 Mar 2010, Robert Haas wrote: On Mon, Mar 29, 2010 at 2:31 PM, Faheem Mitha fah...@email.unc.edu wrote: It's not really too clear to me from reading this what specific questions you're trying to answer. Quote from opt.{tex/pdf}, Section 1: If I have to I can use

[PERFORM] experiments in query optimization

2010-03-25 Thread Faheem Mitha
Hi everyone, I've been trying to reduce both memory usage and runtime for a query. Comments/suggestions gratefully received. Details are at http://bulldog.duhs.duke.edu/~faheem/snppy/opt.pdf See particularly Section 1 - Background and Discussion. If you want a text version, see