Re: [HACKERS] [PERFORM] Releasing memory during External sorting?

2005-09-26 Thread Dann Corbit
-Original Message- From: Ron Peacetree [mailto:[EMAIL PROTECTED] Sent: Saturday, September 24, 2005 3:31 AM To: Dann Corbit; pgsql-hackers@postgresql.org; pgsql- [EMAIL PROTECTED] Subject: RE: [HACKERS] [PERFORM] Releasing memory during External sorting? From: Dann Corbit [EMAIL

Re: [HACKERS] [PERFORM] Releasing memory during External sorting?

2005-09-25 Thread Simon Riggs
On Fri, 2005-09-23 at 12:48 -0400, Ron Peacetree wrote: I have some indications from private tests that very high memory settings may actually hinder performance of the sorts, though I cannot explain that and wonder whether it is the performance tests themselves that have issues. Hmmm.

Re: [HACKERS] [PERFORM] Releasing memory during External sorting?

2005-09-24 Thread Ron Peacetree
From: Dann Corbit [EMAIL PROTECTED] Sent: Sep 23, 2005 5:38 PM Subject: RE: [HACKERS] [PERFORM] Releasing memory during External sorting? _C Unleashed_ also explains how to use a callback function to perform arbitrary radix sorts (you simply need a method that returns the [bucketsize] most

Re: [HACKERS] [PERFORM] Releasing memory during External sorting?

2005-09-23 Thread Ron Peacetree
From: Tom Lane [EMAIL PROTECTED] Sent: Sep 23, 2005 2:15 PM Subject: Re: [PERFORM] Releasing memory during External sorting? Mark Lewis [EMAIL PROTECTED] writes: operations != passes. If you were clever, you could probably write a modified bubble-sort algorithm that only made 2 passes. A

Re: [HACKERS] [PERFORM] Releasing memory during External sorting?

2005-09-23 Thread Ron Peacetree
From: Simon Riggs [EMAIL PROTECTED] Sent: Sep 23, 2005 5:37 AM Subject: [PERFORM] Releasing memory during External sorting? I have concerns about whether we are overallocating memory for use in external sorts. (All code relating to this is in tuplesort.c) A decent external sorting algorithm, say

Re: [HACKERS] [PERFORM] Releasing memory during External sorting?

2005-09-23 Thread Ron Peacetree
Yep. Also, bear in mind that the lg(n!)= ~ nlgn - n lower bound on the number of comparisions: a= says nothing about the amount of data movement used. b= only holds for generic comparison based sorting algorithms. As Knuth says (vol 3, p180), Distribution Counting sorts without ever comparing

Re: [HACKERS] [PERFORM] Releasing memory during External sorting?

2005-09-23 Thread Mark Lewis
operations != passes. If you were clever, you could probably write a modified bubble-sort algorithm that only made 2 passes. A pass is a disk scan, operations are then performed (hopefully in memory) on what you read from the disk. So there's no theoretical log N lower-bound on the number of

Re: [HACKERS] [PERFORM] Releasing memory during External sorting?

2005-09-23 Thread Dann Corbit
- From: [EMAIL PROTECTED] [mailto:pgsql-hackers- [EMAIL PROTECTED] On Behalf Of Ron Peacetree Sent: Friday, September 23, 2005 11:41 AM To: Mark Lewis; Tom Lane; pgsql-hackers@postgresql.org; pgsql- [EMAIL PROTECTED] Subject: Re: [HACKERS] [PERFORM] Releasing memory during External sorting? Yep

Re: [HACKERS] [PERFORM] Releasing memory during External sorting?

2005-09-23 Thread Tom Lane
Ron Peacetree [EMAIL PROTECTED] writes: 2= No optimal external sorting algorithm should use more than 2 passes. 3= Optimal external sorting algorithms should use 1 pass if at all possible. A comparison-based sort must use at least N log N operations, so it would appear to me that if you haven't

Re: [HACKERS] [PERFORM] Releasing memory during External sorting?

2005-09-23 Thread Tom Lane
Mark Lewis [EMAIL PROTECTED] writes: operations != passes. If you were clever, you could probably write a modified bubble-sort algorithm that only made 2 passes. A pass is a disk scan, operations are then performed (hopefully in memory) on what you read from the disk. So there's no

Re: [HACKERS] [PERFORM] Releasing memory during External sorting?

2005-09-23 Thread Dann Corbit
PROTECTED] On Behalf Of Dann Corbit Sent: Friday, September 23, 2005 2:21 PM To: Ron Peacetree; Mark Lewis; Tom Lane; pgsql-hackers@postgresql.org; pgsql-performance@postgresql.org Subject: Re: [HACKERS] [PERFORM] Releasing memory during External sorting? For the subfiles, load the top element