Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-09-15 Thread Robert Haas
On Wed, Sep 15, 2010 at 12:33 PM, Tom Lane wrote: > Robert Haas writes: >> On Wed, Sep 15, 2010 at 12:14 PM, Tom Lane wrote: >>> The above scenario is only a risk if you suppose that dropping a >>> relation that lacks physical storage will nonetheless result in >>> attempted unlink()s.  I think

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-09-15 Thread Tom Lane
Robert Haas writes: > On Wed, Sep 15, 2010 at 12:14 PM, Tom Lane wrote: >> The above scenario is only a risk if you suppose that dropping a >> relation that lacks physical storage will nonetheless result in >> attempted unlink()s.  I think that that's probably not the case; > Why? How would we

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-09-15 Thread Robert Haas
On Wed, Sep 15, 2010 at 12:14 PM, Tom Lane wrote: > Robert Haas writes: >> After further reflection, I think that the above reasoning is wrong. >> GetNewRelFileNode() guarantees that the OID doesn't collide with the >> OID column of pg_class, not the relfilenode column of pg_class; and, >> per th

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-09-15 Thread Tom Lane
Robert Haas writes: > After further reflection, I think that the above reasoning is wrong. > GetNewRelFileNode() guarantees that the OID doesn't collide with the > OID column of pg_class, not the relfilenode column of pg_class; and, > per the comments to that function, it only guarantees this when

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-09-11 Thread Robert Haas
On Thu, Aug 12, 2010 at 1:29 PM, Robert Haas wrote: >> Lastly, it bothers me that you've put in code to delete files belonging >> to temp rels during crash restart, without any code to clean up their >> catalog entries.  This will therefore lead to dangling pg_class >> references, with uncertain b

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-08-13 Thread Tom Lane
Robert Haas writes: > On Fri, Aug 13, 2010 at 12:49 PM, Tom Lane wrote: >> Also, it'd be better if BBB and FFF were marked up as >> rather than , see examples elsewhere in that file. > I see. How should I mark tBBB_FFF? I'd do tBBB_FFF regards, tom lane -- Sent via

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-08-13 Thread Robert Haas
On Fri, Aug 13, 2010 at 12:49 PM, Tom Lane wrote: > Robert Haas writes: >> Here is an updated patch.  It's in context-diff format this time, > > Thanks, I appreciate that ;-) > > This looks committable to me, with a couple of tiny suggestions: Woo hoo! > In the text added to storage.sgml, s/tem

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-08-13 Thread Tom Lane
Robert Haas writes: > Here is an updated patch. It's in context-diff format this time, Thanks, I appreciate that ;-) This looks committable to me, with a couple of tiny suggestions: In the text added to storage.sgml, s/temporary relation/temporary relations/. Also, it'd be better if BBB and FF

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-08-12 Thread Robert Haas
On Thu, Aug 12, 2010 at 2:12 PM, Robert Haas wrote: > That seems like a good idea.  I'll post an updated patch. Here is an updated patch. It's in context-diff format this time, but that made it go over 100K, so I gzip'd it because I can't remember what the attachment size limit is these days. I

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-08-12 Thread Tom Lane
Robert Haas writes: > On Thu, Aug 12, 2010 at 7:04 PM, Tom Lane wrote: >> One other thought about all this: in the past, one objection that's been >> raised to deleting files during crash restart is the possible loss of >> forensic evidence. > ... With this patch, we can be sure of removing ALL

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-08-12 Thread Robert Haas
On Thu, Aug 12, 2010 at 7:04 PM, Tom Lane wrote: > One other thought about all this: in the past, one objection that's been > raised to deleting files during crash restart is the possible loss of > forensic evidence.  It might be a good idea to provide some fairly > simple way for developers to di

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-08-12 Thread Robert Haas
On Thu, Aug 12, 2010 at 6:23 PM, Tom Lane wrote: > Robert Haas writes: >> On Thu, Aug 12, 2010 at 5:29 PM, Alvaro Herrera >> wrote: >>> What about having autovacuum silenty drop the catalog entry if it's a >>> temp entry for which the underlying file does not exist? > >> I think that would be su

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-08-12 Thread Tom Lane
One other thought about all this: in the past, one objection that's been raised to deleting files during crash restart is the possible loss of forensic evidence. It might be a good idea to provide some fairly simple way for developers to disable that deletion subroutine. I'm not sure that it rise

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-08-12 Thread Tom Lane
Robert Haas writes: > On Thu, Aug 12, 2010 at 5:29 PM, Alvaro Herrera > wrote: >> What about having autovacuum silenty drop the catalog entry if it's a >> temp entry for which the underlying file does not exist? > I think that would be subject to race conditions. Well, autovacuum's willingness

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-08-12 Thread Robert Haas
On Thu, Aug 12, 2010 at 5:29 PM, Alvaro Herrera wrote: > Excerpts from Robert Haas's message of jue ago 12 13:29:57 -0400 2010: > >> We have two existing mechanisms for removing the catalog entries: when >> a backend is first asked to access a temporary file, it does a DROP >> SCHEMA ... CASCADE o

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-08-12 Thread Alvaro Herrera
Excerpts from Robert Haas's message of jue ago 12 13:29:57 -0400 2010: > We have two existing mechanisms for removing the catalog entries: when > a backend is first asked to access a temporary file, it does a DROP > SCHEMA ... CASCADE on any pre-existing temp schema. And a table is in > wraparoun

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-08-12 Thread Robert Haas
On Thu, Aug 12, 2010 at 2:06 PM, Tom Lane wrote: > Robert Haas writes: >> On Thu, Aug 12, 2010 at 12:27 PM, Tom Lane wrote: >>> Lastly, it bothers me that you've put in code to delete files belonging >>> to temp rels during crash restart, without any code to clean up their >>> catalog entries.  

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-08-12 Thread Tom Lane
Robert Haas writes: > On Thu, Aug 12, 2010 at 12:27 PM, Tom Lane wrote: >> Lastly, it bothers me that you've put in code to delete files belonging >> to temp rels during crash restart, without any code to clean up their >> catalog entries.  This will therefore lead to dangling pg_class >> referen

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-08-12 Thread Robert Haas
On Thu, Aug 12, 2010 at 12:27 PM, Tom Lane wrote: > Robert Haas writes: >> Here's an updated patch, with the invalidation changes merged in and >> hopefully-suitable adjustments elsewhere. > > I haven't tested this patch, but I read through it (and have I mentioned > how unbelievably illegible -u

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-08-12 Thread Alvaro Herrera
Excerpts from David Fetter's message of jue ago 12 12:59:33 -0400 2010: > On Thu, Aug 12, 2010 at 09:53:54AM -0700, Joshua D. Drake wrote: > > I was under the impression that the project guideline was that we > > only accepted context diffs? > > Since they're trivially producible from unified dif

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-08-12 Thread David Fetter
On Thu, Aug 12, 2010 at 09:53:54AM -0700, Joshua D. Drake wrote: > On Thu, 2010-08-12 at 09:44 -0700, David Fetter wrote: > > On Thu, Aug 12, 2010 at 12:27:45PM -0400, Tom Lane wrote: > > > Robert Haas writes: > > > > Here's an updated patch, with the invalidation changes merged > > > > in and hop

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-08-12 Thread Joshua D. Drake
On Thu, 2010-08-12 at 09:44 -0700, David Fetter wrote: > On Thu, Aug 12, 2010 at 12:27:45PM -0400, Tom Lane wrote: > > Robert Haas writes: > > > Here's an updated patch, with the invalidation changes merged in and > > > hopefully-suitable adjustments elsewhere. > > > > I haven't tested this patch

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-08-12 Thread David Fetter
On Thu, Aug 12, 2010 at 12:27:45PM -0400, Tom Lane wrote: > Robert Haas writes: > > Here's an updated patch, with the invalidation changes merged in and > > hopefully-suitable adjustments elsewhere. > > I haven't tested this patch, but I read through it (and have I mentioned > how unbelievably il

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-08-12 Thread Tom Lane
Robert Haas writes: > Here's an updated patch, with the invalidation changes merged in and > hopefully-suitable adjustments elsewhere. I haven't tested this patch, but I read through it (and have I mentioned how unbelievably illegible -u format patches are?). It seems to be close to commitable,

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-08-06 Thread Alvaro Herrera
Excerpts from Robert Haas's message of vie ago 06 15:32:21 -0400 2010: > > Perhaps run through pg_class after restart and flush anything marked > > relistemp? > > The trouble is that you have to bind to a database before you can run > through pg_class, and the postmaster doesn't. Of course, if i

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-08-06 Thread Robert Haas
On Fri, Aug 6, 2010 at 2:43 PM, Tom Lane wrote: > Robert Haas writes: >> On Fri, Aug 6, 2010 at 2:07 PM, Tom Lane wrote: >>> Sure, it tops out somewhere, but 32K is way too close to configurations >>> we know work well enough in the field (I've seen multiple reports of >>> people using a couple

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-08-06 Thread Robert Haas
On Fri, Aug 6, 2010 at 3:00 PM, Tom Lane wrote: > Robert Haas writes: >> This patch only directly addresses the issue of cleaning up the >> storage, so there are still the catalog entries to worry about.  But >> it doesn't seem impossible to think about building on this approach to >> eventually

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-08-06 Thread Tom Lane
David Fetter writes: > On Fri, Aug 06, 2010 at 03:00:35PM -0400, Tom Lane wrote: >> Perhaps run through pg_class after restart and flush anything marked >> relistemp? Although the ideal solution, probably, would be for temp >> tables to not have persistent catalog entries in the first place. > F

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-08-06 Thread David Fetter
On Fri, Aug 06, 2010 at 03:00:35PM -0400, Tom Lane wrote: > Robert Haas writes: > > This patch only directly addresses the issue of cleaning up the > > storage, so there are still the catalog entries to worry about. > > But it doesn't seem impossible to think about building on this > > approach to

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-08-06 Thread Tom Lane
Robert Haas writes: > This patch only directly addresses the issue of cleaning up the > storage, so there are still the catalog entries to worry about. But > it doesn't seem impossible to think about building on this approach to > eventually handle that part of the problem better, too. I haven't

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-08-06 Thread Tom Lane
Robert Haas writes: > On Fri, Aug 6, 2010 at 2:07 PM, Tom Lane wrote: >> Sure, it tops out somewhere, but 32K is way too close to configurations >> we know work well enough in the field (I've seen multiple reports of >> people using a couple thousand backends). > Well, I wouldn't expect anyone t

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-08-06 Thread Robert Haas
On Fri, Aug 6, 2010 at 2:16 PM, Tom Lane wrote: > Jaime Casanova writes: >> On Fri, Aug 6, 2010 at 12:50 PM, Robert Haas wrote: >>> Just "DROP TABLE pg_temp_2.foo" or whatever and away you go. > >> wow! that's true... and certainly a bug... > > No, it's not a bug.  You'll find only superusers ca

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-08-06 Thread Robert Haas
On Fri, Aug 6, 2010 at 2:07 PM, Tom Lane wrote: > Robert Haas writes: >> On Fri, Aug 6, 2010 at 1:14 PM, Tom Lane wrote: >>> Really?  Surely that should be illegal during normal operation. We >>> might be doing such during crash recovery, but we don't need to >>> broadcast sinval messages then.

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-08-06 Thread Tom Lane
Jaime Casanova writes: > On Fri, Aug 6, 2010 at 12:50 PM, Robert Haas wrote: >> Just "DROP TABLE pg_temp_2.foo" or whatever and away you go. > wow! that's true... and certainly a bug... No, it's not a bug. You'll find only superusers can do it. > we shouldn't allow any session to drop other s

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-08-06 Thread Jaime Casanova
On Fri, Aug 6, 2010 at 12:50 PM, Robert Haas wrote: > > Just "DROP TABLE > pg_temp_2.foo" or whatever and away you go. > wow! that's true... and certainly a bug... we shouldn't allow any session to drop other session's temp tables, or is there a reason for this misbehavior? -- Jaime Casanova   

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-08-06 Thread Tom Lane
Robert Haas writes: > On Fri, Aug 6, 2010 at 1:14 PM, Tom Lane wrote: >> Really?  Surely that should be illegal during normal operation. We >> might be doing such during crash recovery, but we don't need to >> broadcast sinval messages then. > autovacuum.c does it when we start to worry about XI

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-08-06 Thread Robert Haas
On Fri, Aug 6, 2010 at 1:14 PM, Tom Lane wrote: > Robert Haas writes: >> On Thu, Aug 5, 2010 at 7:05 PM, Tom Lane wrote: >>> One thing that I find rather distressing about this is the 25% bloat >>> in sizeof(SharedInvalidationMessage).  Couldn't we avoid that?  Is it >>> really necessary to *eve

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-08-06 Thread Tom Lane
Robert Haas writes: > On Thu, Aug 5, 2010 at 7:05 PM, Tom Lane wrote: >> One thing that I find rather distressing about this is the 25% bloat >> in sizeof(SharedInvalidationMessage).  Couldn't we avoid that?  Is it >> really necessary to *ever* send an SI message for a backend-local rel? > It ca

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-08-06 Thread Robert Haas
On Thu, Aug 5, 2010 at 7:05 PM, Tom Lane wrote: > Robert Haas writes: >> [ BackendRelFileNode patch ] > > One thing that I find rather distressing about this is the 25% bloat > in sizeof(SharedInvalidationMessage).  Couldn't we avoid that?  Is it > really necessary to *ever* send an SI message fo

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-08-05 Thread Tom Lane
Robert Haas writes: > [ BackendRelFileNode patch ] One thing that I find rather distressing about this is the 25% bloat in sizeof(SharedInvalidationMessage). Couldn't we avoid that? Is it really necessary to *ever* send an SI message for a backend-local rel? I agree that one needs to send relc

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-07-28 Thread Jaime Casanova
On Sun, Jul 25, 2010 at 4:32 PM, Robert Haas wrote: > On Sun, Jul 25, 2010 at 2:37 AM, Jaime Casanova wrote: >> but i have a few questions, maybe is right what you did i only want to >> understand it: >> - you added this in include/storage/smgr.h, so why is safe to assume >> that if the backend !

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-07-25 Thread Robert Haas
On Sun, Jul 25, 2010 at 2:37 AM, Jaime Casanova wrote: > but i have a few questions, maybe is right what you did i only want to > understand it: > - you added this in include/storage/smgr.h, so why is safe to assume > that if the backend != InvalidBackendId it must be a temp relation? > > +#define

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-07-24 Thread Jaime Casanova
On Fri, Jul 23, 2010 at 10:05 AM, Jaime Casanova wrote: > On Thu, Jun 10, 2010 at 3:39 PM, Robert Haas wrote: >> >> >> I believe that this patch will clear away one major obstacle to >> implementing global temporary and unlogged tables: it enables us to be >> sure of cleaning up properly after a

Re: [HACKERS] including backend ID in relpath of temp rels - updated patch

2010-07-23 Thread Jaime Casanova
On Thu, Jun 10, 2010 at 3:39 PM, Robert Haas wrote: > > > I believe that this patch will clear away one major obstacle to > implementing global temporary and unlogged tables: it enables us to be > sure of cleaning up properly after a crash without relying on catalog > entries or XLOG.  Based on pr