Re: [HACKERS] small parallel restore optimization

2009-03-09 Thread Andrew Dunstan
Magnus Hagander wrote: Andrew Dunstan wrote: Alvaro Herrera wrote: Andrew Dunstan wrote: I have found the source of the problem I saw. dumputils.c:fmtId() uses a static PQExpBuffer which it initialises the first time it's called. This gets clobbered by simultaneous calls

Re: [HACKERS] small parallel restore optimization

2009-03-09 Thread Magnus Hagander
Andrew Dunstan wrote: > > > Alvaro Herrera wrote: >> Andrew Dunstan wrote: >> >> >>> I have found the source of the problem I saw. dumputils.c:fmtId() >>> uses a static PQExpBuffer which it initialises the first time it's >>> called. This gets clobbered by simultaneous calls by Windows thread

Re: [HACKERS] small parallel restore optimization

2009-03-09 Thread Magnus Hagander
Alvaro Herrera wrote: > Tom Lane wrote: > >> Worst case, we could say that parallel restore isn't supported on >> mingw. I'm not entirely sure why we bother with that platform at all... > > I think you're confusing it with cygwin ... Yeah. Much as I hate working around the quirks of mingw, I de

Re: [HACKERS] small parallel restore optimization

2009-03-08 Thread Alvaro Herrera
Tom Lane wrote: > Worst case, we could say that parallel restore isn't supported on > mingw. I'm not entirely sure why we bother with that platform at all... I think you're confusing it with cygwin ... -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQ

Re: [HACKERS] small parallel restore optimization

2009-03-08 Thread Tom Lane
Andrew Dunstan writes: > Alvaro Herrera wrote: >> Could you use a different static PQExpBuffer on each thread? >> pthread_getspecific sort of thing, only to be used on Windows. > For MSVC we could declare it with "_declspec(thread)" and it would be > allocated in thread-local storage, but it loo

Re: [HACKERS] small parallel restore optimization

2009-03-08 Thread Andrew Dunstan
Alvaro Herrera wrote: Andrew Dunstan wrote: I have found the source of the problem I saw. dumputils.c:fmtId() uses a static PQExpBuffer which it initialises the first time it's called. This gets clobbered by simultaneous calls by Windows threads. I could just make it auto and set it u

Re: [HACKERS] small parallel restore optimization

2009-03-08 Thread Alvaro Herrera
Andrew Dunstan wrote: > I have found the source of the problem I saw. dumputils.c:fmtId() uses a > static PQExpBuffer which it initialises the first time it's called. This > gets clobbered by simultaneous calls by Windows threads. > > I could just make it auto and set it up on each call, but t

Re: [HACKERS] small parallel restore optimization

2009-03-08 Thread Tom Lane
Andrew Dunstan writes: > I have found the source of the problem I saw. dumputils.c:fmtId() uses a > static PQExpBuffer which it initialises the first time it's called. This > gets clobbered by simultaneous calls by Windows threads. Ugh. But that doesn't explain the original trouble report on U

Re: [HACKERS] small parallel restore optimization

2009-03-08 Thread Andrew Dunstan
Tom Lane wrote: I've seen a recent error that suggests we are clobbering memory somewhere in the parallel code, as well as Olivier Prennant's reported error that suggests the same thing, although I'm blessed if I can see where it might be. Maybe some more eyeballs on the code would help.

Re: [HACKERS] small parallel restore optimization

2009-03-07 Thread Andrew Dunstan
Tom Lane wrote: o...@pyrenet.fr writes: the only thing I could come with is a calloc(1,12) that seems to alloc mem for filename, in that case sdewitte.dmp; so the alloc is not counting the null char at the end. Where do you see that? The memtool dump you sent doesn't show it AFAI

Re: [HACKERS] small parallel restore optimization

2009-03-07 Thread Tom Lane
o...@pyrenet.fr writes: > the only thing I could come with is a calloc(1,12) that seems to alloc > mem for filename, in that case sdewitte.dmp; so the alloc is not counting > the null char at the end. Where do you see that? The memtool dump you sent doesn't show it AFAICS.

Re: [HACKERS] small parallel restore optimization

2009-03-06 Thread ohp
On Fri, 6 Mar 2009, Guillaume Smet wrote: Date: Fri, 6 Mar 2009 18:58:58 +0100 From: Guillaume Smet To: Tom Lane Cc: Andrew Dunstan , PostgreSQL-development , o...@pyrenet.fr Subject: Re: [HACKERS] small parallel restore optimization On Fri, Mar 6, 2009 at 6:20 PM, Tom Lane wrote: Can

Re: [HACKERS] small parallel restore optimization

2009-03-06 Thread Guillaume Smet
On Fri, Mar 6, 2009 at 6:20 PM, Tom Lane wrote: > Can you put together even a weakly reproducible test case?  Something > that only fails every tenth or hundredth time would still help. It seems that Olivier can reproduce the problem at will on Unixware. I don't know if it's easy to find useful i

Re: [HACKERS] small parallel restore optimization

2009-03-06 Thread Tom Lane
Andrew Dunstan writes: > Here's a little optimization for the parallel restore code, that > inhibits reopening the archive file unless the worker will actually need > to read from the file (i.e. a data member). It seems to work OK on both > Linux and Windows, and I propose to apply it in a day

[HACKERS] small parallel restore optimization

2009-03-06 Thread Andrew Dunstan
Here's a little optimization for the parallel restore code, that inhibits reopening the archive file unless the worker will actually need to read from the file (i.e. a data member). It seems to work OK on both Linux and Windows, and I propose to apply it in a day or two. I've seen a recent e