Where * ==
{print | save to PDF | save to mumble format | display on screen}
Anyone know of one?
TiA
Ron
---(end of broadcast)---
TIP 3: Have you checked our extensive FAQ?
http://www.postgresql.org/docs/faq
I've now gotten verification from multiple working DBA's that DB2, Oracle, and
SQL Server can achieve ~250MBps ASTR (with as much as ~500MBps ASTR in
setups akin to Oracle RAC) when attached to a decent (not outrageous, but
decent) HD subsystem...
I've not yet had any RW DBA verify Jeff Baker's
First I wanted to verify that pg's IO rates were inferior to The Competition.
Now there's at least an indication that someone else has solved similar
problems. Existence proofs make some things easier ;-)
Is there any detailed programmer level architectual doc set for pg? I know
the best doc is
I'm putting in as much time as I can afford thinking about pg related
performance issues. I'm doing it because of a sincere desire to help
understand and solve them, not to annoy people.
If I didn't believe in pg, I would't be posting thoughts about how to
make it better.
It's probably worth
4, 2005 8:24 AM
To: Simon Riggs [EMAIL PROTECTED]
Cc: Tom Lane [EMAIL PROTECTED], Ron Peacetree [EMAIL PROTECTED],
pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] [PERFORM] A Better External Sort?
On Tue, Oct 04, 2005 at 12:24:54PM +0100, Simon Riggs wrote:
How did you determine the 1500
Jeff, are those _burst_ rates from HD buffer or _sustained_ rates from
actual HD media? Rates from IO subsystem buffer or cache are
usually considerably higher than Average Sustained Transfer Rate.
Also, are you measuring _raw_ HD IO (bits straight off the platters, no
FS or other overhead) or
Let's pretend we get a 24HD HW RAID solution like that J Baker
says he has access to and set it up as a RAID 10. Assuming
it uses two 64b 133MHz PCI-X busses and has the fastest HDs
available on it, Jeff says he can hit ~1GBps of XFS FS IO rate
with that set up (12*83.3MBps= 1GBps).
Josh says
OK, change performance to single thread performance and we
still have a valid starting point for a discussion.
Ron
-Original Message-
From: Gregory Maxwell [EMAIL PROTECTED]
Sent: Oct 3, 2005 8:19 PM
To: Ron Peacetree [EMAIL PROTECTED]
Subject: Re: [HACKERS] [PERFORM] A Better External
*blink* Tapes?! I thought that was a typo...
If our sort is code based on sorting tapes, we've made a mistake. HDs
are not tapes, and Polyphase Merge Sort and it's brethren are not the
best choices for HD based sorts.
Useful references to this point:
Knuth, Vol 3 section 5.4.9, (starts p356 of
-Original Message-
From: Andrew Dunstan [EMAIL PROTECTED]
Sent: Oct 1, 2005 11:19 AM
To: Ron Peacetree [EMAIL PROTECTED]
Subject: Re: [HACKERS] [PERFORM] A Better External Sort?
Ron Peacetree wrote:
The good news is all this means it's easy to demonstrate that we can
improve
@svana.org
Sent: Oct 1, 2005 12:19 PM
Subject: Re: [HACKERS] [PERFORM] A Better External Sort?
On Sat, Oct 01, 2005 at 10:22:40AM -0400, Ron Peacetree wrote:
Assuming we get the abyssmal physical IO performance fixed...
(because until we do, _nothing_ is going to help us as much)
I'm still
than the method you propose.
Ron Peacetree:
1= No that was not my main example. It was the simplest example
used to frame the later more complicated examples. Please don't
get hung up on it.
2= You are incorrect. Since IO is the most expensive operation we
can do, any method that makes two
From: Zeugswetter Andreas DAZ SD [EMAIL PROTECTED]
Sent: Sep 29, 2005 9:28 AM
Subject: RE: [HACKERS] [PERFORM] A Better External Sort?
In my original example, a sequential scan of the 1TB of 2KB
or 4KB records, = 250M or 500M records of data, being sorted
on a binary value key will take ~1000x
From: Josh Berkus josh@agliodbs.com
Sent: Sep 29, 2005 12:54 PM
Subject: Re: [HACKERS] [PERFORM] A Better External Sort?
The biggest single area where I see PostgreSQL external
sort sucking is on index creation on large tables. For
example, for free version of TPCH, it takes only 1.5 hours to
could replace the
present sorting code with infinitely fast sorting code and we'd
still be scrod performance wise.
So why does basic IO suck so badly?
Ron
-Original Message-
From: Josh Berkus josh@agliodbs.com
Sent: Sep 30, 2005 1:23 PM
To: Ron Peacetree [EMAIL PROTECTED]
Cc: pgsql
josh@agliodbs.com
Sent: Sep 30, 2005 4:41 PM
To: Ron Peacetree [EMAIL PROTECTED]
Cc: pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org
Subject: Re: [HACKERS] [PERFORM] A Better External Sort?
Ron,
That 11MBps was your =bulk load= speed. If just loading a table
is this slow
If I've done this correctly, there should not be anywhere near
the number of context switches we currently see while sorting.
Each unscheduled context switch represents something unexpected
occuring or things not being where they are needed when they are
needed. Reducing such circumstances to
From: Jeffrey W. Baker [EMAIL PROTECTED]
Sent: Sep 29, 2005 12:27 AM
To: Ron Peacetree [EMAIL PROTECTED]
Cc: pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org
Subject: Re: [HACKERS] [PERFORM] A Better External Sort?
You are engaging in a length and verbose exercise in mental
From: Jeffrey W. Baker [EMAIL PROTECTED]
Sent: Sep 27, 2005 1:26 PM
To: Ron Peacetree [EMAIL PROTECTED]
Subject: Re: [HACKERS] [PERFORM] A Better External Sort?
On Tue, 2005-09-27 at 13:15 -0400, Ron Peacetree wrote:
That Btree can be used to generate a physical reordering of the data
in one
In the interest of efficiency and not reinventing the wheel, does anyone know
where I can find C or C++ source code for a Btree variant with the following
properties:
A= Data elements (RIDs) are only stored in the leaves, Keys (actually
KeyPrefixes; see D below) and Node pointers are only stored
From: Josh Berkus josh@agliodbs.com
ent: Sep 27, 2005 12:15 PM
To: Ron Peacetree [EMAIL PROTECTED]
Subject: Re: [HACKERS] [PERFORM] A Better External Sort?
I've somehow missed part of this thread, which is a shame since this is
an area of primary concern for me.
Your suggested algorithm seems
From: Dann Corbit [EMAIL PROTECTED]
Sent: Sep 26, 2005 5:13 PM
To: Ron Peacetree [EMAIL PROTECTED], pgsql-hackers@postgresql.org,
pgsql-performance@postgresql.org
Subject: RE: [HACKERS] [PERFORM] A Better External Sort?
I think that the btrees are going to be O(n*log(n)) in construction
SECOND ATTEMPT AT POST. Web mailer appears to have
eaten first one. I apologize in advance if anyone gets two
versions of this post.
=r
From: Tom Lane [EMAIL PROTECTED]
Sent: Sep 26, 2005 9:42 PM
Subject: Re: [HACKERS] [PERFORM] A Better External Sort?
So far, you've blithely assumed that you
From: Ron Peacetree [EMAIL PROTECTED]
Sent: Sep 24, 2005 6:30 AM
Subject: Re: [HACKERS] [PERFORM] Releasing memory during External sorting?
... the amount of IO done is the most
important of the things that you should be optimizing for in
choosing an external sorting algorithm.
snip
Since
From: Dann Corbit [EMAIL PROTECTED]
Sent: Sep 23, 2005 5:38 PM
Subject: RE: [HACKERS] [PERFORM] Releasing memory during External sorting?
_C Unleashed_ also explains how to use a callback function to perform
arbitrary radix sorts (you simply need a method that returns the
[bucketsize] most
From: Tom Lane [EMAIL PROTECTED]
Sent: Sep 23, 2005 2:15 PM
Subject: Re: [PERFORM] Releasing memory during External sorting?
Mark Lewis [EMAIL PROTECTED] writes:
operations != passes. If you were clever, you could probably write a
modified bubble-sort algorithm that only made 2 passes. A
From: Simon Riggs [EMAIL PROTECTED]
Sent: Sep 23, 2005 5:37 AM
Subject: [PERFORM] Releasing memory during External sorting?
I have concerns about whether we are overallocating memory for use in
external sorts. (All code relating to this is in tuplesort.c)
A decent external sorting algorithm, say
in college :)
-- Mark
On Fri, 2005-09-23 at 13:17 -0400, Tom Lane wrote:
Ron Peacetree [EMAIL PROTECTED] writes:
2= No optimal external sorting algorithm should use more than 2 passes.
3= Optimal external sorting algorithms should use 1 pass if at all possible.
A comparison-based sort must use
28 matches
Mail list logo
