Re: "could not reattach to shared memory" on buildfarm member dory

2019-04-08 Thread Noah Misch
On Sun, Apr 07, 2019 at 12:43:23AM -0400, Tom Lane wrote: > Noah Misch writes: > > On Tue, Apr 02, 2019 at 10:09:00AM -0400, Tom Lane wrote: > >> I worry that your proposed fix is unstable, in particular this assumption > >> seems shaky: > >>> + * ... The idea is that, if the allocator handed out

Re: "could not reattach to shared memory" on buildfarm member dory

2019-04-06 Thread Tom Lane
Noah Misch writes: > On Tue, Apr 02, 2019 at 10:09:00AM -0400, Tom Lane wrote: >> I worry that your proposed fix is unstable, in particular this assumption >> seems shaky: >>> + * ... The idea is that, if the allocator handed out >>> + * REGION1 pages before REGION2 pages at one occasion, it will

Re: "could not reattach to shared memory" on buildfarm member dory

2019-04-06 Thread Noah Misch
On Tue, Apr 02, 2019 at 10:09:00AM -0400, Tom Lane wrote: > Noah Misch writes: > > I can reproduce the 4 MiB allocations described > > in https://postgr.es/m/29823.1525132...@sss.pgh.pa.us; a few times per > > "vcregress check", they emerge in the middle of PGSharedMemoryReAttach(). > > The 4 MiB

Re: "could not reattach to shared memory" on buildfarm member dory

2019-04-02 Thread Tom Lane
Noah Misch writes: > I can reproduce the 4 MiB allocations described > in https://postgr.es/m/29823.1525132...@sss.pgh.pa.us; a few times per > "vcregress check", they emerge in the middle of PGSharedMemoryReAttach(). > The 4 MiB allocations are stacks for new threads of the default thread > pool[

Re: "could not reattach to shared memory" on buildfarm member dory

2019-04-02 Thread Noah Misch
On Sun, Dec 02, 2018 at 09:35:06PM -0800, Noah Misch wrote: > On Tue, Sep 25, 2018 at 08:05:12AM -0700, Noah Misch wrote: > > On Mon, Sep 24, 2018 at 01:53:05PM -0400, Tom Lane wrote: > > > Overall, I agree that neither of these approaches are exactly attractive. > > > We're paying a heck of a lot

Re: "could not reattach to shared memory" on buildfarm member dory

2019-01-30 Thread Noah Misch
On Tue, Jan 29, 2019 at 11:28:56AM -0500, Heath Lord wrote: > On Thu, Jan 17, 2019 at 3:27 AM Noah Misch wrote: > > On Sun, Dec 02, 2018 at 09:35:06PM -0800, Noah Misch wrote: > > > Could one of you having a dory login use > > > https://live.sysinternals.com/Procmon.exe to capture process events >

Re: "could not reattach to shared memory" on buildfarm member dory

2019-01-29 Thread Heath Lord
On Thu, Jan 17, 2019 at 3:27 AM Noah Misch wrote: > On Sun, Dec 02, 2018 at 09:35:06PM -0800, Noah Misch wrote: > > On Tue, Sep 25, 2018 at 08:05:12AM -0700, Noah Misch wrote: > > > On Mon, Sep 24, 2018 at 01:53:05PM -0400, Tom Lane wrote: > > > > Overall, I agree that neither of these approaches

Re: "could not reattach to shared memory" on buildfarm member dory

2019-01-17 Thread Noah Misch
On Sun, Dec 02, 2018 at 09:35:06PM -0800, Noah Misch wrote: > On Tue, Sep 25, 2018 at 08:05:12AM -0700, Noah Misch wrote: > > On Mon, Sep 24, 2018 at 01:53:05PM -0400, Tom Lane wrote: > > > Overall, I agree that neither of these approaches are exactly attractive. > > > We're paying a heck of a lot

Re: "could not reattach to shared memory" on buildfarm member dory

2018-09-25 Thread Noah Misch
On Mon, Sep 24, 2018 at 01:53:05PM -0400, Tom Lane wrote: > Noah Misch writes: > > In this proof of concept, the > > postmaster does not close its copy of a backend socket until the backend > > exits. > > That seems unworkable because it would interfere with detection of client > connection drops

Re: "could not reattach to shared memory" on buildfarm member dory

2018-09-24 Thread Tom Lane
Noah Misch writes: > On Tue, May 01, 2018 at 11:31:50AM -0400, Tom Lane wrote: >> Well, at this point the only thing that's entirely clear is that none >> of the ideas I had work. I think we are going to be forced to pursue >> Noah's idea of doing an end-to-end retry. Somebody else will need to

Re: "could not reattach to shared memory" on buildfarm member dory

2018-08-18 Thread Noah Misch
On Tue, May 01, 2018 at 11:31:50AM -0400, Tom Lane wrote: > Well, at this point the only thing that's entirely clear is that none > of the ideas I had work. I think we are going to be forced to pursue > Noah's idea of doing an end-to-end retry. Somebody else will need to > take point on that; I l

Re: "could not reattach to shared memory" on buildfarm member dory

2018-05-01 Thread Tom Lane
Well, at this point the only thing that's entirely clear is that none of the ideas I had work. I think we are going to be forced to pursue Noah's idea of doing an end-to-end retry. Somebody else will need to take point on that; I lack a Windows environment and have already done a lot more blind p

Re: "could not reattach to shared memory" on buildfarm member dory

2018-04-30 Thread Tom Lane
Andres Freund writes: > Heath, could you use process explorer or such to check which processes > are running inside a working backend process? It seems to be possible to enumerate the threads that are present inside a Windows process, although it's not clear to me how much identifying info is ava

Re: "could not reattach to shared memory" on buildfarm member dory

2018-04-30 Thread Thomas Munro
On Tue, May 1, 2018 at 2:59 PM, Noah Misch wrote: > Likely some privileged daemon is creating a thread in every new process. (On > Windows, it's not unusual for one process to create a thread in another > process.) We don't have good control over that. Huh. I was already amazed (as a non-Windo

Re: "could not reattach to shared memory" on buildfarm member dory

2018-04-30 Thread Tom Lane
Noah Misch writes: > On Mon, Apr 30, 2018 at 08:01:40PM -0400, Tom Lane wrote: >> What seems like a plausible theory at this point is that the apparent >> asynchronicity is due to the allocation being triggered by a different >> thread, and the fact that our added monitoring code seems to make the

Re: "could not reattach to shared memory" on buildfarm member dory

2018-04-30 Thread Noah Misch
On Mon, Apr 30, 2018 at 08:01:40PM -0400, Tom Lane wrote: > It's clear from dory's results that something is causing a 4MB chunk > of memory to get reserved in the process's address space, sometimes. > It might happen during the main MapViewOfFileEx call, or during the > preceding VirtualFree, or w

Re: "could not reattach to shared memory" on buildfarm member dory

2018-04-30 Thread Andres Freund
Him On 2018-04-30 20:01:40 -0400, Tom Lane wrote: > What seems like a plausible theory at this point is that the apparent > asynchronicity is due to the allocation being triggered by a different > thread, and the fact that our added monitoring code seems to make the > failure more likely can be ex

Re: "could not reattach to shared memory" on buildfarm member dory

2018-04-30 Thread Tom Lane
I wrote: > The solution I was thinking about last night was to have > PGSharedMemoryReAttach call MapViewOfFileEx to map the shared memory > segment at an unspecified address, then unmap it, then call VirtualFree, > and finally call MapViewOfFileEx with the real target address. The idea > here is

Re: "could not reattach to shared memory" on buildfarm member dory

2018-04-30 Thread Tom Lane
Heath Lord writes: >So what I noticed after adding the '--force' flag was that in the Event > Viewer logs there was an Error in the System log stating that "The > application-specific permission settings do not grant Local Activation > permission for the COM Server application" for the Runtime

Re: "could not reattach to shared memory" on buildfarm member dory

2018-04-30 Thread Stephen Frost
Greetings, * Heath Lord (heath.l...@crunchydata.com) wrote: >Any ideas or changes that we could do to help debug or verify would be > helpful. We have considered changing it to run everything as SYSTEM but if > possible we would like to avoid this for security reasons. Thank you in > advance

Re: "could not reattach to shared memory" on buildfarm member dory

2018-04-30 Thread Heath Lord
On Mon, Apr 30, 2018 at 2:34 PM, Stephen Frost wrote: > Greetings, > > * Tom Lane (t...@sss.pgh.pa.us) wrote: > > [ Thanks to Stephen for cranking up a continuous build loop on dory ] > > That was actually Heath, who is also trying to work this issue and has > an idea about something else which m

Re: "could not reattach to shared memory" on buildfarm member dory

2018-04-30 Thread Stephen Frost
Greetings, * Tom Lane (t...@sss.pgh.pa.us) wrote: > [ Thanks to Stephen for cranking up a continuous build loop on dory ] That was actually Heath, who is also trying to work this issue and has an idea about something else which might help (and some more information about what's happening in the e

Re: "could not reattach to shared memory" on buildfarm member dory

2018-04-30 Thread Tom Lane
[ Thanks to Stephen for cranking up a continuous build loop on dory ] Noah Misch writes: > On Tue, Apr 24, 2018 at 11:37:33AM +1200, Thomas Munro wrote: >> Maybe try asking what's mapped there with VirtualQueryEx() on failure? > +1. An implementation of that: > https://www.postgresql.org/messag

Re: "could not reattach to shared memory" on buildfarm member dory

2018-04-28 Thread Stephen Frost
Greetings, * Tom Lane (t...@sss.pgh.pa.us) wrote: > Noah Misch writes: > > On Tue, Apr 24, 2018 at 11:37:33AM +1200, Thomas Munro wrote: > >> Maybe try asking what's mapped there with VirtualQueryEx() on failure? > > > +1. An implementation of that: > > https://www.postgresql.org/message-id/201

Re: "could not reattach to shared memory" on buildfarm member dory

2018-04-27 Thread Tom Lane
Noah Misch writes: > On Tue, Apr 24, 2018 at 11:37:33AM +1200, Thomas Munro wrote: >> Maybe try asking what's mapped there with VirtualQueryEx() on failure? > +1. An implementation of that: > https://www.postgresql.org/message-id/20170403065106.GA2624300%40tornado.leadboat.com Not seeing any ot

Re: "could not reattach to shared memory" on buildfarm member dory

2018-04-27 Thread Stephen Frost
Greetings, * Magnus Hagander (mag...@hagander.net) wrote: > On Tue, Apr 24, 2018 at 1:18 AM, Stephen Frost wrote: > > I've asked Heath to take a look at the system again and see if there's > > any Windows logs or such that might help us understand what's happening. > > AV was disabled on the box,

Re: "could not reattach to shared memory" on buildfarm member dory

2018-04-26 Thread Craig Ringer
On 24 April 2018 at 15:18, Magnus Hagander wrote: > Back when I was combating windows AV on a daily basis, this normally did not > have the same effect. Just disabling the AV didn't actually remove the parts > that caused issues, it just hid them. Actual uninstall is what was required. Yep. Spec

Re: "could not reattach to shared memory" on buildfarm member dory

2018-04-24 Thread Magnus Hagander
On Tue, Apr 24, 2018 at 1:18 AM, Stephen Frost wrote: > Greetings, > > * Tom Lane (t...@sss.pgh.pa.us) wrote: > > So far, dory has failed three times with essentially identical symptoms: > > > > 2018-04-23 19:57:10.624 GMT [2240] FATAL: could not reattach to shared > memory (key=0190

Re: "could not reattach to shared memory" on buildfarm member dory

2018-04-24 Thread Noah Misch
On Tue, Apr 24, 2018 at 11:37:33AM +1200, Thomas Munro wrote: > On Tue, Apr 24, 2018 at 11:18 AM, Stephen Frost wrote: > > Greetings, > > > > * Tom Lane (t...@sss.pgh.pa.us) wrote: > >> So far, dory has failed three times with essentially identical symptoms: > >> > >> 2018-04-23 19:57:10.624 GMT [

Re: "could not reattach to shared memory" on buildfarm member dory

2018-04-23 Thread Thomas Munro
On Tue, Apr 24, 2018 at 11:18 AM, Stephen Frost wrote: > Greetings, > > * Tom Lane (t...@sss.pgh.pa.us) wrote: >> So far, dory has failed three times with essentially identical symptoms: >> >> 2018-04-23 19:57:10.624 GMT [2240] FATAL: could not reattach to shared >> memory (key=0190,

Re: "could not reattach to shared memory" on buildfarm member dory

2018-04-23 Thread Stephen Frost
Greetings, * Tom Lane (t...@sss.pgh.pa.us) wrote: > So far, dory has failed three times with essentially identical symptoms: > > 2018-04-23 19:57:10.624 GMT [2240] FATAL: could not reattach to shared > memory (key=0190, addr=018E): error code 487 > 2018-04-23 15:57:10.65

"could not reattach to shared memory" on buildfarm member dory

2018-04-23 Thread Tom Lane
So far, dory has failed three times with essentially identical symptoms: 2018-04-23 19:57:10.624 GMT [2240] FATAL: could not reattach to shared memory (key=0190, addr=018E): error code 487 2018-04-23 15:57:10.657 EDT [8836] ERROR: lost connection to parallel worker 2018-