Re: [HACKERS] Attempt to stop dead instance can stop a random process?

2007-08-31 Thread Tom Lane
[EMAIL PROTECTED] writes: > Hm. I've come to expect the OS removing all pidfiles early at bootup. If there's a script in your system that does that, then adding Postgres lockfiles to it makes all kinds of sense. Our problem as upstream software is that this isn't something well-standardized that

Re: [HACKERS] Attempt to stop dead instance can stop a random process?

2007-08-31 Thread tomas
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Fri, Aug 31, 2007 at 02:41:47PM -0500, Kevin Grittner wrote: [...] > > The real question there is how come the postmaster died without removing > > the pidfile. It's not that easy to crash the postmaster ... > > Well, that's not due to a bug in

Re: [HACKERS] [PATCH] Lazy xid assingment V2

2007-08-31 Thread August Zajonc
Florian G. Pflug wrote: You could still leak them (ie, you move to final location and bail before commit) but it seems to narrow the window down significantly. That leak could be prevented I think if the COMMIT record indicated which files are to be moved, and the actual move happens after the f

Re: [HACKERS] [PATCH] Lazy xid assingment V2

2007-08-31 Thread Alvaro Herrera
Tom Lane wrote: > I note that pg_description defines xid_age as "age of a transaction ID, > in transactions before current transaction", and I'm content to stick > with that definition. Even if it does force consuming an XID. I'm not > sure that we could devise a comparably succinct description

Re: [HACKERS] [PATCH] Lazy xid assingment V2

2007-08-31 Thread Florian G. Pflug
Tom Lane wrote: "Florian G. Pflug" <[EMAIL PROTECTED]> writes: Actually, after more thought on this I'd argue that the only really correct reference for computing a xid's age is GetOldestXmin(). It's what VACUUM uses to calculate the freezeLimit, after all. Now, using GetOldestXmin() directly is

Re: [HACKERS] [PATCH] Lazy xid assingment V2

2007-08-31 Thread Tom Lane
"Florian G. Pflug" <[EMAIL PROTECTED]> writes: > Hm.. but xid_age already exposes an implementation detail (that xid > comparison is circular). But the only reason anyone cares about it at all, really, is that xid comparison is circular ... the function was only invented in the first place to help

Re: [HACKERS] [PATCH] Lazy xid assingment V2

2007-08-31 Thread Florian G. Pflug
Alvaro Herrera wrote: Florian G. Pflug wrote: August Zajonc wrote: I assume that you meant that mail to go to pgsql-hackers too, and just clicked the wrong button by accident. If not, please forgive that I post the reply to the list. Are you going to look at putting temp relations into a specia

Re: [HACKERS] [PATCH] Lazy xid assingment V2

2007-08-31 Thread Florian G. Pflug
Tom Lane wrote: "Florian G. Pflug" <[EMAIL PROTECTED]> writes: Tom Lane wrote: I don't much like this, since as I mentioned before I don't think MyProc->xmin is going to be constant over a whole transaction for long. I don't think xid_age is performance-critical in any way, so I'd vote for let

Re: [HACKERS] [PATCH] Lazy xid assingment V2

2007-08-31 Thread Alvaro Herrera
Florian G. Pflug wrote: > August Zajonc wrote: > I assume that you meant that mail to go to pgsql-hackers too, and just > clicked the wrong button by accident. If not, please forgive that > I post the reply to the list. > >> Are you going to look at putting temp relations into a special folder for

Re: [HACKERS] [PATCH] Lazy xid assingment V2

2007-08-31 Thread Florian G. Pflug
Gregory Stark wrote: "Tom Lane" <[EMAIL PROTECTED]> writes: "Florian G. Pflug" <[EMAIL PROTECTED]> writes: Tom Lane wrote: I don't much like this, since as I mentioned before I don't think MyProc->xmin is going to be constant over a whole transaction for long. I don't think xid_age is perfor

Re: [HACKERS] [PATCH] Lazy xid assingment V2

2007-08-31 Thread Florian G. Pflug
August Zajonc wrote: I assume that you meant that mail to go to pgsql-hackers too, and just clicked the wrong button by accident. If not, please forgive that I post the reply to the list. Are you going to look at putting temp relations into a special folder for 8.4? I really like that idea for c

Re: [HACKERS] [PATCH] Lazy xid assingment V2

2007-08-31 Thread Tom Lane
Gregory Stark <[EMAIL PROTECTED]> writes: > It seems both this and some of the other cases of having to call > ReadNewTransactionId would be eliminated if we invented a new xid treated > similarly to current_time() and others which are calculated once per > transaction and then cached for subsequen

Re: [HACKERS] pgsql: Apply a band-aid fix for the problem that 8.2 and up completely

2007-08-31 Thread Tom Lane
Gregory Stark <[EMAIL PROTECTED]> writes: > "Tom Lane" <[EMAIL PROTECTED]> writes: >> Log Message: >> --- >> Apply a band-aid fix for the problem that 8.2 and up completely misestimate >> the number of rows likely to be produced by a query such as >> SELECT * FROM t1 LEFT JOIN t2 USING (key

Re: [HACKERS] pgsql: Apply a band-aid fix for the problem that 8.2 and up completely

2007-08-31 Thread Gregory Stark
"Tom Lane" <[EMAIL PROTECTED]> writes: > Log Message: > --- > Apply a band-aid fix for the problem that 8.2 and up completely misestimate > the number of rows likely to be produced by a query such as > SELECT * FROM t1 LEFT JOIN t2 USING (key) WHERE t2.key IS NULL; I'm a little war

Re: [HACKERS] [PATCH] Lazy xid assingment V2

2007-08-31 Thread Gregory Stark
"Tom Lane" <[EMAIL PROTECTED]> writes: > "Florian G. Pflug" <[EMAIL PROTECTED]> writes: >> Tom Lane wrote: >>> I don't much like this, since as I mentioned before I don't think >>> MyProc->xmin is going to be constant over a whole transaction for >>> long. I don't think xid_age is performance-cri

Re: [HACKERS] [PATCH] Lazy xid assingment V2

2007-08-31 Thread Tom Lane
"Florian G. Pflug" <[EMAIL PROTECTED]> writes: > Tom Lane wrote: >> I don't much like this, since as I mentioned before I don't think >> MyProc->xmin is going to be constant over a whole transaction for >> long. I don't think xid_age is performance-critical in any way, >> so I'd vote for letting i

Re: [HACKERS] [PATCH] Lazy xid assingment V2

2007-08-31 Thread Florian G. Pflug
Tom Lane wrote: "Florian G. Pflug" <[EMAIL PROTECTED]> writes: A few comments ... GetOldestXmin() is changed slightly - it used to use GetTopTransactionId() as a starting point for it's xmin calculation, falling back to ReadNewTransactionId() if called from outside a transaction. Now, it always

Re: [HACKERS] [GENERAL] Undetected corruption of table files

2007-08-31 Thread Decibel!
On Fri, Aug 31, 2007 at 03:11:29PM -0400, Tom Lane wrote: > Decibel! <[EMAIL PROTECTED]> writes: > > Even if we don't care about folks running on suspect hardware, having a > > CRC would make it far more reasonable to recommend full_page_writes=3Doff. > > This argument seems ridiculous. Finding o

Re: [HACKERS] [PATCH] Lazy xid assingment V2

2007-08-31 Thread Tom Lane
"Florian G. Pflug" <[EMAIL PROTECTED]> writes: > The patch can be found at: http://soc.phlo.org/lazyxidassign.v2.patch > (Seems that large attachments don't get through on this list - or was that > just bad luck when I tried posting the first version of that patch?) No, the message size limit on -

Re: [HACKERS] Attempt to stop dead instance can stop a random process?

2007-08-31 Thread Kevin Grittner
>>> On Fri, Aug 31, 2007 at 3:10 PM, in message <[EMAIL PROTECTED]>, Tom Lane <[EMAIL PROTECTED]> wrote: > "Kevin Grittner" <[EMAIL PROTECTED]> writes: >> Well, that's not due to a bug in PostgreSQL. We're using a buggy LDAP >> implementation (not my call) which can crash things. The machine to

Re: [HACKERS] Password requirement in windows installer

2007-08-31 Thread Andrew Sullivan
On Fri, Aug 31, 2007 at 08:20:20PM +0100, Gregory Stark wrote: > Except note that ident is, like X, precisely the kind of protocol where the > handshake matters least. Since you all the relevant data comes early in the > message you can fire the SYN and the ACK (with the predicted sequence number)

Re: [HACKERS] Attempt to stop dead instance can stop a random process?

2007-08-31 Thread Tom Lane
"Kevin Grittner" <[EMAIL PROTECTED]> writes: > Well, that's not due to a bug in PostgreSQL. We're using a buggy LDAP > implementation (not my call) which can crash things. The machine totally > locked up after logging distress messages from that daemon, and they cycled > power to get out of it.

Re: [HACKERS] Attempt to stop dead instance can stop a random process?

2007-08-31 Thread Kevin Grittner
>>> On Fri, Aug 31, 2007 at 2:18 PM, in message <[EMAIL PROTECTED]>, Tom Lane <[EMAIL PROTECTED]> wrote: > "Kevin Grittner" <[EMAIL PROTECTED]> writes: >> It appears that when pg_ctl gets a stop request for a given directory, it l= >> ooks for a pid file in that directory and signals that pid to

[HACKERS] [PATCH] Lazy xid assingment V2

2007-08-31 Thread Florian G. Pflug
Hi I've updated the lazy xid assignment patch, incorporating the results of the discussion over the past few days. I'll try to explain what the patch does, and why. This reasoning got a bit lengthy, but I felt that I should summarize my own thought, plus the relevant parts of the discussions here

Re: [HACKERS] Password requirement in windows installer

2007-08-31 Thread Andrew Sullivan
On Fri, Aug 31, 2007 at 02:12:03PM -0500, Decibel! wrote: > ISTM that if someone breaches your network to the point where they can > spoof identd, you're pretty much hosed anyway; so what's the point of > hard-coding passwords in a config file somewhere then? True. I personally prefer cryptograph

Re: [HACKERS] Password requirement in windows installer

2007-08-31 Thread Gregory Stark
"Andrew Sullivan" <[EMAIL PROTECTED]> writes: > On Fri, Aug 31, 2007 at 07:07:40PM +0100, Gregory Stark wrote: >> > >> > It shouldn't be easy. Ident uses TCP, which is rather harder to >> > spoof. >> >> Say what? It's actually quite easy to spoof TCP. There are even command-line >> tools to do

Re: [HACKERS] Attempt to stop dead instance can stop a random process?

2007-08-31 Thread Tom Lane
"Kevin Grittner" <[EMAIL PROTECTED]> writes: > It appears that when pg_ctl gets a stop request for a given directory, it l= > ooks for a pid file in that directory and signals that pid to stop. It doe= > sn't appear to check that the pid is for a PostgreSQL postmaster running ou= > t of the given

Re: [HACKERS] Password requirement in windows installer

2007-08-31 Thread Decibel!
On Fri, Aug 31, 2007 at 02:38:25PM -0400, Andrew Sullivan wrote: > Indeed, I would argue that, for industrial-class data centre use, if > you can't use ident between machines, your network security is in > very bad shape. (That isn't to say I think it's a good idea; but > rather, that I hope the n

Re: [HACKERS] [GENERAL] Undetected corruption of table files

2007-08-31 Thread Tom Lane
Decibel! <[EMAIL PROTECTED]> writes: > Even if we don't care about folks running on suspect hardware, having a > CRC would make it far more reasonable to recommend full_page_writes=3Doff. This argument seems ridiculous. Finding out that you have corrupt data is no substitute for not having corrup

Re: [HACKERS] enum types and binary queries

2007-08-31 Thread Andrew Dunstan
Andrew Dunstan wrote: This is a furfy that has been raised before and explained before. Of course, as usual I misspelled it, the word is "furphy". I didn't realise that it was an Australianism. It means more or less "a red herring". Wikipedia says that it is a term particularly popular in

Re: [HACKERS] Password requirement in windows installer

2007-08-31 Thread Andrew Sullivan
On Fri, Aug 31, 2007 at 07:07:40PM +0100, Gregory Stark wrote: > > > > It shouldn't be easy. Ident uses TCP, which is rather harder to > > spoof. > > Say what? It's actually quite easy to spoof TCP. There are even command-line > tools to do it available in most Unix distributions. Sorry, I sho

Re: [HACKERS] enum types and binary queries

2007-08-31 Thread Andrew Dunstan
Decibel! wrote: On Fri, Aug 31, 2007 at 01:41:47PM -0400, Andrew Dunstan wrote: The technique of using a lookup table that you seem to refer to doesn't need any special support from the catalogs or the type system. It's used today in millions of applications. But it can have quite a high c

Re: [HACKERS] Password requirement in windows installer

2007-08-31 Thread Gregory Stark
"Andrew Sullivan" <[EMAIL PROTECTED]> writes: > On Fri, Aug 31, 2007 at 12:30:02PM -0500, Decibel! wrote: >> >> Is it easy to spoof where an incoming connection request is coming from? >> Is there something else that makes ident on 127.0.0.1/32 insecure? > > It shouldn't be easy. Ident uses TCP,

Re: [HACKERS] enum types and binary queries

2007-08-31 Thread Decibel!
On Fri, Aug 31, 2007 at 01:41:47PM -0400, Andrew Dunstan wrote: > The technique of using a lookup table that you seem to refer to doesn't > need any special support from the catalogs or the type system. It's used > today in millions of applications. But it can have quite a high cost in > extra j

Re: [HACKERS] Password requirement in windows installer

2007-08-31 Thread Andrew Sullivan
On Fri, Aug 31, 2007 at 12:30:02PM -0500, Decibel! wrote: > > Is it easy to spoof where an incoming connection request is coming from? > Is there something else that makes ident on 127.0.0.1/32 insecure? It shouldn't be easy. Ident uses TCP, which is rather harder to spoof. If someone can origi

Re: [HACKERS] enum types and binary queries

2007-08-31 Thread Andrew Dunstan
Decibel! wrote: The last time I worked on a project where we had C code access the database, we added stuff to map C enums to ints in the database (along with a parent table to store the enum label). ISTM it'd be good if we had a way to get the numeric index out of an enum. If you mean here

Re: [HACKERS] Password requirement in windows installer

2007-08-31 Thread Decibel!
On Fri, Aug 31, 2007 at 12:37:16PM -0400, Andrew Dunstan wrote: > > > Decibel! wrote: > >Is there something insecure about using ident sameuser for localhost > >authentication on Windows? > > > > FWIW, I never advise people to use ident auth for postgres except on > local (a.k.a. Unix domain

Re: [HACKERS] enum types and binary queries

2007-08-31 Thread Decibel!
On Fri, Aug 31, 2007 at 08:49:05AM -0400, Merlin Moncure wrote: > On 8/30/07, Tom Lane <[EMAIL PROTECTED]> wrote: > > "Merlin Moncure" <[EMAIL PROTECTED]> writes: > > > I noticed that enums are not available to be queried as binary through > > > the protocol. > > > > What do you think the binary re

[HACKERS] Attempt to stop dead instance can stop a random process?

2007-08-31 Thread Kevin Grittner
It appears that when pg_ctl gets a stop request for a given directory, it looks for a pid file in that directory and signals that pid to stop. It doesn't appear to check that the pid is for a PostgreSQL postmaster running out of the given directory. I think it should, although on a quick scan

Re: [HACKERS] Password requirement in windows installer

2007-08-31 Thread Andrew Dunstan
Decibel! wrote: Is there something insecure about using ident sameuser for localhost authentication on Windows? FWIW, I never advise people to use ident auth for postgres except on local (a.k.a. Unix domain socket) connections, which don't exist on Windows. cheers andrew -

Re: [HACKERS] Final background writer cleanup for 8.3

2007-08-31 Thread Greg Smith
On Fri, 31 Aug 2007, Jan Wieck wrote: Again, the original theory for the bgwriter wasn't moving writes out of the critical path, but smoothing responsetimes that tended to go completely down the toilet during checkpointing, causing all the users to wake up and overload the system entirely. A

Re: [HACKERS] Password requirement in windows installer

2007-08-31 Thread Dave Page
Decibel! wrote: > On Fri, Aug 31, 2007 at 09:02:49AM +0100, Dave Page wrote: >> Decibel! wrote: >>> Why does the windows installer require a password for the superuser >>> account, since it's perfectly legitimate not to have a password on that >>> account? I could see perhaps producing a warning, b

Re: [HACKERS] [GENERAL] Undetected corruption of table files

2007-08-31 Thread Decibel!
On Fri, Aug 31, 2007 at 02:34:09PM +0200, Albe Laurenz wrote: > I have thought some more about it, and tend to agree now: > Checksums will only detect disk failure, and that's only > one of the many integrity problems that can happen. > And one that can be reduced to a reasonable degree with good >

Re: [HACKERS] Password requirement in windows installer

2007-08-31 Thread Decibel!
On Fri, Aug 31, 2007 at 09:02:49AM +0100, Dave Page wrote: > Decibel! wrote: > > Why does the windows installer require a password for the superuser > > account, since it's perfectly legitimate not to have a password on that > > account? I could see perhaps producing a warning, but making this a ha

Re: [HACKERS] enum types and binary queries

2007-08-31 Thread Andrew Dunstan
Gregory Stark wrote: enum OIDs are unique across enums? This seems like a strange way to do it. I recall conversations about this a while back though and there were limitations of the type system that led to this, right? No, not the type system as such. It stems from this quote from Tom:

Re: [HACKERS] enum types and binary queries

2007-08-31 Thread Tom Lane
Gregory Stark <[EMAIL PROTECTED]> writes: > enum OIDs are unique across enums? This seems like a strange way to do it. That decision was already made, we are not revisiting it (at least not for 8.3). regards, tom lane ---(end of broadcast)-

Re: [HACKERS] enum types and binary queries

2007-08-31 Thread Gregory Stark
"Alvaro Herrera" <[EMAIL PROTECTED]> writes: > Gregory Stark escribió: > >> I think it would be ok only if a pg_dump/pg_restore reliably restored the >> same >> oid->enum value mapping. Otherwise a binary dump is useless. But as I >> understand it that's the case currently, is it? er, lost a "n

Re: [HACKERS] enum types and binary queries

2007-08-31 Thread Alvaro Herrera
Gregory Stark escribió: > I think it would be ok only if a pg_dump/pg_restore reliably restored the same > oid->enum value mapping. Otherwise a binary dump is useless. But as I > understand it that's the case currently, is it? That doesn't work if the dump is restored on a database that already h

Re: [HACKERS] enum types and binary queries

2007-08-31 Thread Gregory Stark
"Merlin Moncure" <[EMAIL PROTECTED]> writes: > On 8/30/07, Tom Lane <[EMAIL PROTECTED]> wrote: >> "Merlin Moncure" <[EMAIL PROTECTED]> writes: >> > I noticed that enums are not available to be queried as binary through >> > the protocol. >> >> What do you think the binary representation ought to b

Re: [HACKERS] enum types and binary queries

2007-08-31 Thread Tom Lane
"Merlin Moncure" <[EMAIL PROTECTED]> writes: > On 8/30/07, Tom Lane <[EMAIL PROTECTED]> wrote: >> What do you think the binary representation ought to be? Copying OIDs >> seems pretty useless. > I actually think this would be ok, if you mean pg_enum.oid, or the > string would be fine too. I agre

Re: [HACKERS] enum types and binary queries

2007-08-31 Thread Andrew Dunstan
Merlin Moncure wrote: One other very small observation: afaict, there appears to be no way to list enum contents in psql (although you can list the enums themselves in the type listing). select enum_range(null::myenumtype); I'm willing to take a stab at these things if Andrew is busy.

Re: [HACKERS] enum types and binary queries

2007-08-31 Thread Merlin Moncure
On 8/30/07, Tom Lane <[EMAIL PROTECTED]> wrote: > "Merlin Moncure" <[EMAIL PROTECTED]> writes: > > I noticed that enums are not available to be queried as binary through > > the protocol. > > What do you think the binary representation ought to be? Copying OIDs > seems pretty useless. I'm on the

Re: [HACKERS] Final background writer cleanup for 8.3

2007-08-31 Thread Jan Wieck
On 8/24/2007 8:41 AM, Heikki Linnakangas wrote: If anyone out there has a repeatable test case where bgwriter does help, I'm all ears. The theory of moving the writes out of the critical path does sound reasonable, so I'm sure there is test case to demonstrate the effect, but it seems to be prett

Re: [HACKERS] Performing antijoin in postgres

2007-08-31 Thread Suresh_
Thanks for the reply. I can do this at the higher level; but this antijoin thing is a small part of a bigger project. So I want to do it internally in one of the joins. Does anyone have experience in storing tupleslpts and comparing them ?? Gregory Stark wrote: > > "Suresh_" <[EMAIL PROTECTED

Re: [HACKERS] Final background writer cleanup for 8.3

2007-08-31 Thread Jan Wieck
On 8/24/2007 1:17 AM, Greg Smith wrote: On Thu, 23 Aug 2007, Tom Lane wrote: It is doubtless true in a lightly loaded system, but once the kernel is under any kind of memory pressure I think it's completely wrong. The fact that so many tests I've done or seen get maximum throughput in terms

Re: [HACKERS] [GENERAL] Undetected corruption of table files

2007-08-31 Thread Albe Laurenz
Jan Wieck wrote: > Computing a checksum just before writing the block will NOT detect any > faulty memory or Postgres bug that corrupted the block. You will have a > perfectly fine checksum over the corrupted data. > > A checksum only detects corruptions that happen between write and read. > Mo

Re: [HACKERS] [GENERAL] Undetected corruption of table files

2007-08-31 Thread Jan Wieck
On 8/28/2007 4:14 AM, Albe Laurenz wrote: Not all databases are on enterprise scale storage systems, and there's also the small possibility of PostgreSQL bugs that could be detected that way. Computing a checksum just before writing the block will NOT detect any faulty memory or Postgres bug t

Re: [HACKERS] Performing antijoin in postgres

2007-08-31 Thread Gregory Stark
"Suresh_" <[EMAIL PROTECTED]> writes: > I want to add an antijoin operator to PostgreSql. I think you can already do what you want like this: SELECT * FROM A FULL OUTER JOIN B ON (x=y) WHERE x IS NULL OR y IS NULL So for example: postgres=# select * from a; i --- 1 2 (2 rows) pos

Re: [HACKERS] Performing antijoin in postgres

2007-08-31 Thread Heikki Linnakangas
Suresh_ wrote: > I want to add an antijoin operator to PostgreSql. Basically I want to store > the result (tupletableslot) of one join and then compare it with > another..How do I store these results and how do I compare them ? I would suggest implementing it at a higher level, rewriting a query

[HACKERS] Performing antijoin in postgres

2007-08-31 Thread Suresh_
I want to add an antijoin operator to PostgreSql. Basically I want to store the result (tupletableslot) of one join and then compare it with another..How do I store these results and how do I compare them ? -- View this message in context: http://www.nabble.com/Performing-antijoin-in-postgres-

Re: [HACKERS] Password requirement in windows installer

2007-08-31 Thread Dave Page
Decibel! wrote: > Why does the windows installer require a password for the superuser > account, since it's perfectly legitimate not to have a password on that > account? I could see perhaps producing a warning, but making this a hard > requirement seems like overkill. Security out of the box. The