Re: [HACKERS] [PERFORM] Releasing memory during External sorting?

2005-09-26 Thread Dann Corbit
> -Original Message- > From: Ron Peacetree [mailto:[EMAIL PROTECTED] > Sent: Saturday, September 24, 2005 3:31 AM > To: Dann Corbit; pgsql-hackers@postgresql.org; pgsql- > [EMAIL PROTECTED] > Subject: RE: [HACKERS] [PERFORM] Releasing memory during External sortin

Re: [HACKERS] [PERFORM] Releasing memory during External sorting?

2005-09-25 Thread Simon Riggs
On Fri, 2005-09-23 at 12:48 -0400, Ron Peacetree wrote: > > I have some indications from private tests that very high memory settings > >may actually hinder performance of the sorts, though I cannot explain that > >and wonder whether it is the performance tests themselves that have issues. > > > H

Re: [HACKERS] [PERFORM] Releasing memory during External sorting?

2005-09-24 Thread Ron Peacetree
From: Dann Corbit <[EMAIL PROTECTED]> Sent: Sep 23, 2005 5:38 PM Subject: RE: [HACKERS] [PERFORM] Releasing memory during External sorting? >_C Unleashed_ also explains how to use a callback function to perform >arbitrary radix sorts (you simply need a method that returns the >[b

Re: [HACKERS] [PERFORM] Releasing memory during External sorting?

2005-09-23 Thread Dann Corbit
t; [EMAIL PROTECTED] On Behalf Of Dann Corbit > Sent: Friday, September 23, 2005 2:21 PM > To: Ron Peacetree; Mark Lewis; Tom Lane; pgsql-hackers@postgresql.org; > pgsql-performance@postgresql.org > Subject: Re: [HACKERS] [PERFORM] Releasing memory during External sorting? > > For

Re: [HACKERS] [PERFORM] Releasing memory during External sorting?

2005-09-23 Thread Dann Corbit
--Original Message- > From: [EMAIL PROTECTED] [mailto:pgsql-hackers- > [EMAIL PROTECTED] On Behalf Of Ron Peacetree > Sent: Friday, September 23, 2005 11:41 AM > To: Mark Lewis; Tom Lane; pgsql-hackers@postgresql.org; pgsql- > [EMAIL PROTECTED] > Subject: Re: [HACKERS] [PERFO

Re: [HACKERS] [PERFORM] Releasing memory during External sorting?

2005-09-23 Thread Mark Lewis
operations != passes. If you were clever, you could probably write a modified bubble-sort algorithm that only made 2 passes. A pass is a disk scan, operations are then performed (hopefully in memory) on what you read from the disk. So there's no theoretical log N lower-bound on the number of dis

Re: [HACKERS] [PERFORM] Releasing memory during External sorting?

2005-09-23 Thread Ron Peacetree
Yep. Also, bear in mind that the lg(n!)= ~ nlgn - n lower bound on the number of comparisions: a= says nothing about the amount of data movement used. b= only holds for generic comparison based sorting algorithms. As Knuth says (vol 3, p180), Distribution Counting sorts without ever comparing ele

Re: [HACKERS] [PERFORM] Releasing memory during External sorting?

2005-09-23 Thread Ron Peacetree
From: Simon Riggs <[EMAIL PROTECTED]> Sent: Sep 23, 2005 5:37 AM Subject: [PERFORM] Releasing memory during External sorting? >I have concerns about whether we are overallocating memory for use in >external sorts. (All code relating to this is in tuplesort.c) > A decent external sorting algorithm,

Re: [HACKERS] [PERFORM] Releasing memory during External sorting?

2005-09-23 Thread Ron Peacetree
From: Tom Lane <[EMAIL PROTECTED]> Sent: Sep 23, 2005 2:15 PM Subject: Re: [PERFORM] Releasing memory during External sorting? >Mark Lewis <[EMAIL PROTECTED]> writes: >> operations != passes. If you were clever, you could probably write a >> modified bubble-sort algorithm that only made 2 passes

Re: [HACKERS] [PERFORM] Releasing memory during External sorting?

2005-09-23 Thread Tom Lane
Mark Lewis <[EMAIL PROTECTED]> writes: > operations != passes. If you were clever, you could probably write a > modified bubble-sort algorithm that only made 2 passes. A pass is a > disk scan, operations are then performed (hopefully in memory) on what > you read from the disk. So there's no the

Re: [HACKERS] [PERFORM] Releasing memory during External sorting?

2005-09-23 Thread Tom Lane
Ron Peacetree <[EMAIL PROTECTED]> writes: > 2= No optimal external sorting algorithm should use more than 2 passes. > 3= Optimal external sorting algorithms should use 1 pass if at all possible. A comparison-based sort must use at least N log N operations, so it would appear to me that if you have