Re: [HACKERS] Reducing the overhead of NUMERIC data

2005-11-03 Thread Tom Lane
Gregory Maxwell <[EMAIL PROTECTED]> writes: > Another way to look at this is in the context of compression: With > unicode, characters are really 32bit values... But only a small range > of these values is common. So we store and work with them in a > compressed format, UTF-8. > As such it might

Re: slru.c race condition (was Re: [HACKERS] TRAP: FailedAssertion("!((itemid)->lp_flags

2005-11-03 Thread Tom Lane
"Jim C. Nasby" <[EMAIL PROTECTED]> writes: > Seriously, I am wondering about the performance hit of always checking > debug_assertions. > http://archives.postgresql.org/pgsql-hackers/2005-08/msg00389.php > indicates that even with debug_assertions=false, --enable-cassert has a > big performance imp

Re: [HACKERS] Reducing the overhead of NUMERIC data

2005-11-03 Thread Gregory Maxwell
On 11/3/05, Martijn van Oosterhout wrote: > That's called UTF-16 and is currently not supported by PostgreSQL at > all. That may change, since the locale library ICU requires UTF-16 for > everything. UTF-16 doesn't get us out of the variable length character game, for that we need UTF-32... Unles

Re: slru.c race condition (was Re: [HACKERS] TRAP: FailedAssertion("!((itemid)->lp_flags

2005-11-03 Thread Jim C. Nasby
On Thu, Nov 03, 2005 at 11:11:42AM -0500, Tom Lane wrote: > Perhaps rather than an all-or-nothing debug_assertions GUC variable, > what we want is something that turns on or off "expensive" assertion > checks at runtime. This could include MEMORY_CONTEXT_CHECKING and > anything else where the actu

Re: [HACKERS] Reducing the overhead of NUMERIC data

2005-11-03 Thread Simon Riggs
On Thu, 2005-11-03 at 10:32 -0500, Tom Lane wrote: > Bruce Momjian writes: > >> On Thu, 3 Nov 2005, Simon Riggs wrote: > >>> At the moment we've established we can do this fairly much for free. > > > Agreed. With the proposal, we are saving perhaps 5% storage space for > > numeric fields, but ar

Re: [HACKERS] Exclusive lock for database rename

2005-11-03 Thread Andrew - Supernews
On 2005-11-03, Peter Eisentraut <[EMAIL PROTECTED]> wrote: > Peter Eisentraut wrote: >> Someone wanted to rename a database while someone else was running a >> rather long pg_dump, so the rename had to wait, and everyone else had >> to wait for the rename because no new connections would be allowed

Re: [HACKERS] Reducing the overhead of NUMERIC data

2005-11-03 Thread Andrew - Supernews
On 2005-11-03, Martijn van Oosterhout wrote: >> For "other databases", the column could be encoded as 2 byte characters >> or 4 byte characters, allowing it to be fixed. I find myself doubting >> that ASCII characters could be encoded more efficiently in such formats, >> than the inclusion of a le

Re: [HACKERS] Spinlocks, yet again: analysis and proposed patches

2005-11-03 Thread Simon Riggs
On Thu, 2005-11-03 at 08:03 -0800, Mark Wong wrote: > On Tue, 01 Nov 2005 07:32:32 + > Simon Riggs <[EMAIL PROTECTED]> wrote: > > Concerned about the awful checkpointing. Can you bump wal_buffers to > > 8192 just to make sure? Thats way too high, but just to prove it. > > > > We need to rdeuce

Re: [HACKERS] Reducing the overhead of NUMERIC data

2005-11-03 Thread Martijn van Oosterhout
On Thu, Nov 03, 2005 at 12:28:02PM -0500, [EMAIL PROTECTED] wrote: > It's unfortunate that the length is encoded multiple times. In UTF-8, > for instance, each character has its length encoded in the most > significant bits. Complicated to extract, however, the data is encoded > twice. 1 in the hea

Re: [HACKERS] Reducing the overhead of NUMERIC data

2005-11-03 Thread Simon Riggs
On Thu, 2005-11-03 at 11:36 -0500, Andrew Dunstan wrote: > Well, it could also be argued that DW apps could often get away with > using floating point types, even where the primary source needs to be in > fixed point for accuracy, and that could generate lots of savings in > speed and space. B

Re: [HACKERS] [COMMITTERS] pgsql: Rename the members of CommandDest enum so

2005-11-03 Thread Tom Lane
Alvaro Herrera <[EMAIL PROTECTED]> writes: > Marc G. Fournier wrote: >> Is this an appropriate commit for post-Beta? Does it fix a critical bug >> that will prevent a release from happening? :( > Well, I asked -hackers before committing, and there were no objections, > plus there were a couple o

Re: [HACKERS] Exclusive lock for database rename

2005-11-03 Thread Tom Lane
Peter Eisentraut <[EMAIL PROTECTED]> writes: > It's clear that we need to prevent new connections to the database > being renamed, but isn't there a less intrusive way to do that (row > locks?)? This is fixed already for 8.1. regards, tom lane --

Re: [HACKERS] Reducing the overhead of NUMERIC data

2005-11-03 Thread mark
On Thu, Nov 03, 2005 at 03:09:26PM +0100, Martijn van Oosterhout wrote: > On Thu, Nov 03, 2005 at 01:49:46PM +, Simon Riggs wrote: > > In other databases, CHAR(12) and NUMERIC(12) are fixed length datatypes. > > In PostgreSQL, they are dynamically varying datatypes. > Please explain how a CHAR(

Re: [HACKERS] Exclusive lock for database rename

2005-11-03 Thread Peter Eisentraut
Peter Eisentraut wrote: > Someone wanted to rename a database while someone else was running a > rather long pg_dump, so the rename had to wait, and everyone else had > to wait for the rename because no new connections would be allowed. As an auxiliary issue, why do the new connections have to wai

Re: [HACKERS] [COMMITTERS] pgsql: Rename the members of CommandDest enum so

2005-11-03 Thread Alvaro Herrera
Marc G. Fournier wrote: > > Is this an appropriate commit for post-Beta? Does it fix a critical bug > that will prevent a release from happening? :( Well, I asked -hackers before committing, and there were no objections, plus there were a couple of messages that weren't actual encouragements bu

[HACKERS] Exclusive lock for database rename

2005-11-03 Thread Peter Eisentraut
Someone wanted to rename a database while someone else was running a rather long pg_dump, so the rename had to wait, and everyone else had to wait for the rename because no new connections would be allowed. It's clear that we need to prevent new connections to the database being renamed, but i

Re: [HACKERS] Reducing the overhead of NUMERIC data

2005-11-03 Thread Andrew Dunstan
Simon Riggs wrote: On Wed, 2005-11-02 at 19:12 -0500, Tom Lane wrote: Andrew Dunstan <[EMAIL PROTECTED]> writes: Could someone please quantify how much bang we might get for what seems like quite a lot of bucks? I appreciate the need for speed, but the saving here strikes me as marg

Re: [HACKERS] Reducing the overhead of NUMERIC data

2005-11-03 Thread Simon Riggs
On Thu, 2005-11-03 at 07:03 -0800, Stephan Szabo wrote: > I don't believe the above is safe to say, yet. AFAICS, this has been > discussed only on hackers (and patches) in this discussion, whereas this > sort of change should probably be brought up on general as well to get a > greater understandi

Re: slru.c race condition (was Re: [HACKERS] TRAP: FailedAssertion("!((itemid)->lp_flags

2005-11-03 Thread Tom Lane
Alvaro Herrera <[EMAIL PROTECTED]> writes: > May I propose to make Assert() yield only WARNINGs, That is a horrid idea --- for one thing, it means that Asserts inside the elog machinery itself would be instant infinite recursion, and even elsewhere you'd have to think a bit about whether it's ok t

Re: [HACKERS] Spinlocks, yet again: analysis and proposed patches

2005-11-03 Thread Mark Wong
On Tue, 01 Nov 2005 07:32:32 + Simon Riggs <[EMAIL PROTECTED]> wrote: > On Mon, 2005-10-31 at 16:10 -0800, Mark Wong wrote: > > On Thu, 20 Oct 2005 23:03:47 +0100 > > Simon Riggs <[EMAIL PROTECTED]> wrote: > > > > > On Wed, 2005-10-19 at 14:07 -0700, Mark Wong wrote: > > > > > > > > > > This

Re: slru.c race condition (was Re: [HACKERS] TRAP: FailedAssertion("!((itemid)->lp_flags

2005-11-03 Thread Alvaro Herrera
Jim C. Nasby wrote: > BTW, is MEMORY_CONTEXT_CHECKING that expensive? It seems like it > shouldn't be, but I'm only guessing at what exactly it does... Yes, because not only it checks marker bytes at the end of palloc chunks and similar stuff, but it also overwrites whole contexts with 0x7f when

Re: [HACKERS] Reducing the overhead of NUMERIC data

2005-11-03 Thread Tom Lane
Bruce Momjian writes: >> On Thu, 3 Nov 2005, Simon Riggs wrote: >>> At the moment we've established we can do this fairly much for free. > Agreed. With the proposal, we are saving perhaps 5% storage space for > numeric fields, but are adding code complexity and reducing its possible > precision.

Re: [HACKERS] Reducing the overhead of NUMERIC data

2005-11-03 Thread Bruce Momjian
Stephan Szabo wrote: > > On Thu, 3 Nov 2005, Simon Riggs wrote: > > > On Wed, 2005-11-02 at 19:12 -0500, Tom Lane wrote: > > > Andrew Dunstan <[EMAIL PROTECTED]> writes: > > > > Could someone please quantify how much bang we might get for what seems > > > > like quite a lot of bucks? > > > > I ap

Re: [HACKERS] Reducing the overhead of NUMERIC data

2005-11-03 Thread Marcus Engene
Simon Riggs wrote: On Thu, 2005-11-03 at 11:13 -0300, Alvaro Herrera wrote: Simon Riggs wrote: On PostgreSQL, CHAR(12) is a bpchar datatype with all instantiations of that datatype having a 4 byte varlena header. In this example, all of those instantiations having the varlena header set to 12

Re: [HACKERS] Reducing the overhead of NUMERIC data

2005-11-03 Thread Stephan Szabo
On Thu, 3 Nov 2005, Simon Riggs wrote: > On Wed, 2005-11-02 at 19:12 -0500, Tom Lane wrote: > > Andrew Dunstan <[EMAIL PROTECTED]> writes: > > > Could someone please quantify how much bang we might get for what seems > > > like quite a lot of bucks? > > > I appreciate the need for speed, but the

Re: [HACKERS] Reducing the overhead of NUMERIC data

2005-11-03 Thread Simon Riggs
On Thu, 2005-11-03 at 11:13 -0300, Alvaro Herrera wrote: > Simon Riggs wrote: > > On PostgreSQL, CHAR(12) is a bpchar datatype with all instantiations of > > that datatype having a 4 byte varlena header. In this example, all of > > those instantiations having the varlena header set to 12, so essent

Re: [HACKERS] Reducing the overhead of NUMERIC data

2005-11-03 Thread Alvaro Herrera
Simon Riggs wrote: > On PostgreSQL, CHAR(12) is a bpchar datatype with all instantiations of > that datatype having a 4 byte varlena header. In this example, all of > those instantiations having the varlena header set to 12, so essentially > wasting the 4 byte header. We need the length word beca

Re: [HACKERS] Reducing the overhead of NUMERIC data

2005-11-03 Thread Martijn van Oosterhout
On Thu, Nov 03, 2005 at 01:49:46PM +, Simon Riggs wrote: > In other databases, CHAR(12) and NUMERIC(12) are fixed length datatypes. > In PostgreSQL, they are dynamically varying datatypes. Please explain how a CHAR(12) can store 12 UTF-8 characters when each character may be 1 to 4 bytes, unle

[HACKERS] Transitive closure of a directed graph

2005-11-03 Thread Steinar H. Gunderson
Hi, I was asked to post this here for any interested parties -- please Cc me on any comments/followups as I'm not subscribed to -hackers. This is a sort-of writeup on how to find the transitive closure[1] of a directed graph G(V,E), or G+(V,E) if you want to get into notation. (There are multipl

Re: [HACKERS] Reducing the overhead of NUMERIC data

2005-11-03 Thread Simon Riggs
On Thu, 2005-11-03 at 08:27 +, Simon Riggs wrote: > On Wed, 2005-11-02 at 19:12 -0500, Tom Lane wrote: > > If we were willing to invent the "varlena2" datum format then we could > > save four bytes per numeric, plus reduce numeric's alignment requirement > > from int to short which would probab

Re: [HACKERS] Reducing the overhead of NUMERIC data

2005-11-03 Thread Simon Riggs
On Wed, 2005-11-02 at 19:12 -0500, Tom Lane wrote: > Andrew Dunstan <[EMAIL PROTECTED]> writes: > > Could someone please quantify how much bang we might get for what seems > > like quite a lot of bucks? > > I appreciate the need for speed, but the saving here strikes me as > > marginal at best, u