Re: Tom Lane 2015-05-29 <13871.1432921...@sss.pgh.pa.us>
> Why can't the user stop it? We won't be bleating about the case of a
> symlink to a non-writable file someplace else, which is the Debian use
> case. I don't see a very good excuse to have a non-writable file right
> in the data directory
* Andres Freund (and...@anarazel.de) wrote:
> On 2015-05-29 13:49:16 -0400, Tom Lane wrote:
> > > That sounds like a potentially nontrivial amount of repetitive log bleat
> > > after every crash start? One which the user can't really stop?
> >
> > Why can't the user stop it?
>
> Because it makes
On 2015-05-29 14:15:48 -0400, Tom Lane wrote:
> Andres Freund writes:
> > On 2015-05-29 13:49:16 -0400, Tom Lane wrote:
> >> Why can't the user stop it?
>
> > Because it makes a good amount of sense to have e.g. certificates not
> > owned by postgres and not writeable? You don't necessarily want
Andres Freund writes:
> On 2015-05-29 13:49:16 -0400, Tom Lane wrote:
>> Why can't the user stop it?
> Because it makes a good amount of sense to have e.g. certificates not
> owned by postgres and not writeable? You don't necessarily want to
> symlink them somewhere else, because that makes movin
On 2015-05-29 13:49:16 -0400, Tom Lane wrote:
> Andres Freund writes:
> > On 2015-05-29 13:14:18 -0400, Tom Lane wrote:
> >> Abhijit Menon-Sen writes:
> >> As I mentioned yesterday, I'm not really on board with ignoring EACCES,
> >> except for the directories-on-Windows case. Since we're only lo
Andres Freund writes:
> On 2015-05-29 13:14:18 -0400, Tom Lane wrote:
>> Abhijit Menon-Sen writes:
>> As I mentioned yesterday, I'm not really on board with ignoring EACCES,
>> except for the directories-on-Windows case. Since we're only logging
>> the failures anyway, I think it is reasonable t
On 2015-05-29 13:14:18 -0400, Tom Lane wrote:
> Abhijit Menon-Sen writes:
> As I mentioned yesterday, I'm not really on board with ignoring EACCES,
> except for the directories-on-Windows case. Since we're only logging
> the failures anyway, I think it is reasonable to log a complaint for any
> u
At 2015-05-29 13:14:18 -0400, t...@sss.pgh.pa.us wrote:
>
> Pushed with minor revisions.
Thanks, looks good.
> Since we're only logging the failures anyway, I think it is reasonable
> to log a complaint for any unwritable file in the data directory.
Sounds reasonable, patch attached. ETXTBSY ha
Abhijit Menon-Sen writes:
> At 2015-05-28 17:37:16 -0400, t...@sss.pgh.pa.us wrote:
>> I have to leave shortly, so I'll look at the initdb cleanup tomorrow.
> Here's a revision of that patch that's more along the lines of what you
> committed.
Pushed with minor revisions.
> It occurred to me th
Michael Paquier writes:
> On Fri, May 29, 2015 at 6:49 PM, Amit Kapila wrote:
>> Test - 2 - Directory Symlink for pg_xlog
>> First 4 steps are same as Test-1
>> 5. mklink /D E:\ PGData\pg_xlog E:\TempLog\xlog_symlink_loc
>> 6. Restart Server - Error
>> - FATAL: required WAL directory "pg_xlog"
On Fri, May 29, 2015 at 6:49 PM, Amit Kapila wrote:
> On Fri, May 29, 2015 at 2:28 PM, Michael Paquier
> wrote:
>>
>> On Fri, May 29, 2015 at 5:01 PM, Amit Kapila
>> wrote:
>> >
>> > Test-3 - Symlinks in pg_tblspc.
>> > 1. Create couple of tablespaces which creates symlinks
>> > in pg_tblspc
>>
Re: Tom Lane 2015-05-28 <5740.1432849...@sss.pgh.pa.us>
> Abhijit Menon-Sen writes:
> > Here's an updated patch for the fsync problem(s).
>
> I've committed this after some mostly-cosmetic rearrangements.
Fwiw, I can confirm that the problem is fixed for 9.5. The regression
tests I've added to p
On Fri, May 29, 2015 at 2:28 PM, Michael Paquier
wrote:
>
> On Fri, May 29, 2015 at 5:01 PM, Amit Kapila
wrote:
> >
> > Test-3 - Symlinks in pg_tblspc.
> > 1. Create couple of tablespaces which creates symlinks
> > in pg_tblspc
> > 2. Crash the server
> > 3. Restart Server - It works fine.
> >
>
On Fri, May 29, 2015 at 5:01 PM, Amit Kapila wrote:
> On Fri, May 29, 2015 at 9:25 AM, Tom Lane wrote:
>>
>>
>> Speaking of which, could somebody test that the committed patch does
>> what it's supposed to on Windows? You were worried upthread about
>> whether the tests for symlinks (aka junctio
On Fri, May 29, 2015 at 9:25 AM, Tom Lane wrote:
>
>
> Speaking of which, could somebody test that the committed patch does
> what it's supposed to on Windows? You were worried upthread about
> whether the tests for symlinks (aka junction points) behaved correctly,
> and I have no way to check th
On Fri, May 29, 2015 at 12:55 PM, Tom Lane wrote:
> Abhijit Menon-Sen writes:
>>> I have to leave shortly, so I'll look at the initdb cleanup tomorrow.
>
>> Here's a revision of that patch that's more along the lines of what you
>> committed.
>
> Will look at that tomorrow ...
>
>> It occurred to
Abhijit Menon-Sen writes:
>> I have to leave shortly, so I'll look at the initdb cleanup tomorrow.
> Here's a revision of that patch that's more along the lines of what you
> committed.
Will look at that tomorrow ...
> It occurred to me that we should probably also silently skip EACCES on
> ope
At 2015-05-28 17:37:16 -0400, t...@sss.pgh.pa.us wrote:
>
> I've committed this after some mostly-cosmetic rearrangements.
Thank you.
> I have to leave shortly, so I'll look at the initdb cleanup tomorrow.
Here's a revision of that patch that's more along the lines of what you
committed.
It occ
Abhijit Menon-Sen writes:
> Here's an updated patch for the fsync problem(s).
I've committed this after some mostly-cosmetic rearrangements.
> 4. As a partial aside, why does fsync_fname use OpenTransientFile? It
>looks like it should use BasicOpenFile as pre_sync_fname does. We
>close t
On Thu, May 28, 2015 at 1:04 PM, Tom Lane wrote:
> Robert Haas writes:
>> On Thu, May 28, 2015 at 10:26 AM, Abhijit Menon-Sen
>> wrote:
>>> 2. Robert, are you comfortable with what fsync_pgdata() does in xlog.c?
>>> I wasn't sure if I should move that to fd.c as well. I think it's
>>> borderlin
At 2015-05-28 19:56:15 +0530, a...@2ndquadrant.com wrote:
>
> I have a separate patch to initdb with the corresponding changes, which
> I will post after dinner and a bit more testing.
Here's that patch too.
I was a bit unsure how far to go with this. It fixes the problem of not
following pg_xlog
Robert Haas writes:
> On Thu, May 28, 2015 at 10:26 AM, Abhijit Menon-Sen
> wrote:
>> 2. Robert, are you comfortable with what fsync_pgdata() does in xlog.c?
>> I wasn't sure if I should move that to fd.c as well. I think it's
>> borderline OK for now.
> I think if the function is specific as f
On Thu, May 28, 2015 at 10:26 AM, Abhijit Menon-Sen
wrote:
> 2. Robert, are you comfortable with what fsync_pgdata() does in xlog.c?
>I wasn't sure if I should move that to fd.c as well. I think it's
>borderline OK for now.
I think if the function is specific as fsync_pgdata(), fd.c is n
Hi.
Here's an updated patch for the fsync problem(s).
A few points that may be worth considering:
1. I've made the ReadDir → ReadDirExtended change, but in retrospect I
don't think it's really worth it. Using ReadDir and letting it die
is probably fine. (Aside: I left ReadDirExtended stati
At 2015-05-27 23:41:29 -0400, t...@sss.pgh.pa.us wrote:
>
> What about turning ReadDir into a wrapper around a ReadDirExtended
> function that adds an elevel argument?
Sounds reasonable, doing.
-- Abhijit
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to
Abhijit Menon-Sen writes:
> At 2015-05-27 20:22:18 -0400, t...@sss.pgh.pa.us wrote:
>> I doubt that that (not using AllocateDir) [â¦]
> (Disregarding per your followup.)
>> What I think is a reasonable compromise is to treat AllocateDir
>> failure as nonfatal, but to continue using ReadDir etc
At 2015-05-27 20:22:18 -0400, t...@sss.pgh.pa.us wrote:
>
> I doubt that that (not using AllocateDir) […]
(Disregarding per your followup.)
> What I think is a reasonable compromise is to treat AllocateDir
> failure as nonfatal, but to continue using ReadDir etc which means
> that any post-open f
I wrote:
> Abhijit Menon-Sen writes:
>> At 2015-05-27 11:46:39 +0530, a...@2ndquadrant.com wrote:
>>> I'm trying a couple of approaches to that (e.g. using readdir directly
>>> instead of ReadDir), but other suggestions are welcome.
>> Here's what that looks like, but not yet fully tested.
> I d
Abhijit Menon-Sen writes:
> At 2015-05-27 11:46:39 +0530, a...@2ndquadrant.com wrote:
>> I'm trying a couple of approaches to that (e.g. using readdir directly
>> instead of ReadDir), but other suggestions are welcome.
> Here's what that looks like, but not yet fully tested.
I doubt that that (n
At 2015-05-27 11:46:39 +0530, a...@2ndquadrant.com wrote:
>
> I'm trying a couple of approaches to that (e.g. using readdir directly
> instead of ReadDir), but other suggestions are welcome.
Here's what that looks like, but not yet fully tested.
-- Abhijit
diff --git a/src/backend/access/transam/
At 2015-05-26 22:44:03 +0200, and...@anarazel.de wrote:
>
> So what I propose is:
> 1) Remove the automatic symlink following
> 2) Follow pg_tbspc/*, pg_xlog if it's a symlink, fix the latter in
>initdb -S
> 3) Add a elevel argument to walkdir(), return if AllocateDir() fails,
>continue for
At 2015-05-26 22:44:03 +0200, and...@anarazel.de wrote:
>
> So, this was discussed in the following thread, starting at:
> http://archives.postgresql.org/message-id/20150403163232.GA28444%40eldon.alvh.no-ip.org
Sorry, I didn't see this before replying.
> There are no other places we it's "allowed
At 2015-05-26 19:07:20 +0200, and...@anarazel.de wrote:
>
> Abhijit, do you recall why the code was changed to follow all symlinks
> in contrast to explicitly going through the tablespaces as initdb -S
> does? I'm pretty sure early versions of the patch pretty much had a
> verbatim copy of the init
On 2015-05-26 19:07:20 +0200, Andres Freund wrote:
> It is somewhat interesting that similar code has been used in
> pg_upgrade, via initdb -S, for a while now, without, to my knowledge, it
> causing reported problem. I think the relevant difference is that that
> code doesn't follow symlinks. It'
On 2015-05-26 19:07:20 +0200, Andres Freund wrote:
> Abhijit, do you recall why the code was changed to follow all symlinks
> in contrast to explicitly going through the tablespaces as initdb -S
> does? I'm pretty sure early versions of the patch pretty much had a
> verbatim copy of the initdb logi
On 2015-05-26 10:41:12 -0400, Tom Lane wrote:
> Yeah. Perhaps I missed it, but was the original patch motivated by
> actual failures that had been seen in the field, or was it just a
> hypothetical concern?
I'd mentioned that it might be a good idea to do this while
investingating a bug with unlo
Andrew Dunstan writes:
> On 05/26/2015 11:58 AM, Tom Lane wrote:
>> One thing perhaps we *should* be selective about, though, is which
>> symlinks we try to follow. I think that a good case could be made
>> for ignoring symlinks everywhere except in the pg_tablespace directory.
>> If we did, that
On Tue, May 26, 2015 at 6:16 PM, Andrew Dunstan wrote:
>
> On 05/26/2015 11:58 AM, Tom Lane wrote:
>
>> Andrew Dunstan writes:
>>
>>> OK, I'm late to the party. But why exactly are we syncing absolutely
>>> everything? That seems over-broad.
>>>
>> If we try to be selective, we risk errors of om
On 05/26/2015 11:58 AM, Tom Lane wrote:
Andrew Dunstan writes:
OK, I'm late to the party. But why exactly are we syncing absolutely
everything? That seems over-broad.
If we try to be selective, we risk errors of omission, which no one would
ever notice until someone's data got eaten in a low-
Andrew Dunstan writes:
> OK, I'm late to the party. But why exactly are we syncing absolutely
> everything? That seems over-broad.
If we try to be selective, we risk errors of omission, which no one would
ever notice until someone's data got eaten in a low-probability crash
scenario. It seems m
On 05/26/2015 08:05 AM, Robert Haas wrote:
On Mon, May 25, 2015 at 9:54 PM, Stephen Frost wrote:
I certainly see your point, but Tom also pointed out that it's not great
to ignore failures during this phase:
* Tom Lane (t...@sss.pgh.pa.us) wrote:
Greg Stark writes:
What exactly is failing?
Robert Haas writes:
> Anything we do short of making all errors in this area non-fatal is
> going to leave behind startup-failure cases that exist today, and we
> have no evidence at this time that such startup failures would be
> justified by any actual data loss risk.
Yeah. Perhaps I missed it
On Mon, May 25, 2015 at 9:54 PM, Andres Freund wrote:
> On 2015-05-25 21:33:03 -0400, Robert Haas wrote:
>> On Mon, May 25, 2015 at 2:20 PM, Tom Lane wrote:
>> > Perhaps, but if we didn't have permission to write the file, it's hard to
>> > argue that it's our responsibility to fsync it. So this
On Mon, May 25, 2015 at 9:54 PM, Stephen Frost wrote:
> I certainly see your point, but Tom also pointed out that it's not great
> to ignore failures during this phase:
>
> * Tom Lane (t...@sss.pgh.pa.us) wrote:
>> Greg Stark writes:
>> > What exactly is failing?
>> > Is it that fsync is returnin
* Abhijit Menon-Sen (a...@2ndquadrant.com) wrote:
> At 2015-05-26 03:54:51 +0200, and...@anarazel.de wrote:
> > Another thing is whether we should handle a recursive symlink in
> > pgdata? I personally think not, but...
>
> I think not too.
Yikes.. That's definitely the kind of thing that's why
At 2015-05-26 03:54:51 +0200, and...@anarazel.de wrote:
>
> Say a symlink goes to a binary, which is currently being executed:
> ETXTBSY. Or the file is in a readonly filesystem: EROFS. So we'd
> need to ignore a lot of errors, possibly ignoring valid ones.
Right. That's why I started out by bein
On 2015-05-25 21:33:03 -0400, Robert Haas wrote:
> On Mon, May 25, 2015 at 2:20 PM, Tom Lane wrote:
> > Perhaps, but if we didn't have permission to write the file, it's hard to
> > argue that it's our responsibility to fsync it. So this seems like it's
> > adding complexity without really adding
Robert,
* Robert Haas (robertmh...@gmail.com) wrote:
> On Mon, May 25, 2015 at 2:20 PM, Tom Lane wrote:
>> Andres Freund writes:
>>> On 2015-05-25 14:14:10 -0400, Stephen Frost wrote:
Not really sure I see that as helping.
>>> On most OSs, except windows and some obscure unixes, a readonly
On Mon, May 25, 2015 at 2:20 PM, Tom Lane wrote:
> Andres Freund writes:
>> On 2015-05-25 14:14:10 -0400, Stephen Frost wrote:
Additionally we could attempt to fsync with a readonly fd before trying
the read-write fd...
>
>>> Not really sure I see that as helping.
>
>> On most OSs, exce
On 05/25/2015 01:21 PM, Tom Lane wrote:
> And from the other direction, where exactly is it written that
> distros/users will only create problematic files at the top level of
> $PGDATA? I'd have zero confidence in such an assertion applied to
> tablespace directories, for sure.
Yes, absolutely.
Alvaro Herrera writes:
> Tom Lane wrote:
>> Well, that opens us to errors of omission, ie failing to fsync things we
>> should have. Maybe that's an okay risk, but personally I'd judge that
>> "fsync everything and ignore (some?) errors" is probably a more robust
>> approach over time.
> How is
Tom Lane wrote:
> Stephen Frost writes:
> > I've not followed this thread all that closely, but I do tend to agree
> > with the idea of "only try to mess with files that are *clearly* ours to
> > mess with."
>
> Well, that opens us to errors of omission, ie failing to fsync things we
> should hav
Andres Freund writes:
> On 2015-05-25 14:14:10 -0400, Stephen Frost wrote:
>>> Additionally we could attempt to fsync with a readonly fd before trying
>>> the read-write fd...
>> Not really sure I see that as helping.
> On most OSs, except windows and some obscure unixes, a readonly fd is
> allo
On 2015-05-25 14:02:28 -0400, Tom Lane wrote:
> Stephen Frost writes:
> > I've not followed this thread all that closely, but I do tend to agree
> > with the idea of "only try to mess with files that are *clearly* ours to
> > mess with."
>
> Well, that opens us to errors of omission, ie failing to
* Andres Freund (and...@anarazel.de) wrote:
> On 2015-05-25 14:14:10 -0400, Stephen Frost wrote:
> > That seems overly complicated, for my 2c at least. I don't particularly
> > like trying to mess with files that might be rightfully considered "not
> > ours" either.
>
> I'd not consider an fsync
* Andres Freund (and...@anarazel.de) wrote:
> On 2015-05-25 14:02:28 -0400, Tom Lane wrote:
> > Stephen Frost writes:
> > > I've not followed this thread all that closely, but I do tend to agree
> > > with the idea of "only try to mess with files that are *clearly* ours to
> > > mess with."
> >
>
Stephen Frost writes:
> I've not followed this thread all that closely, but I do tend to agree
> with the idea of "only try to mess with files that are *clearly* ours to
> mess with."
Well, that opens us to errors of omission, ie failing to fsync things we
should have. Maybe that's an okay risk,
On 2015-05-25 14:14:10 -0400, Stephen Frost wrote:
> That seems overly complicated, for my 2c at least. I don't particularly
> like trying to mess with files that might be rightfully considered "not
> ours" either.
I'd not consider an fsync to be "messing" with files, especially if
they're in PGD
* Andres Freund (and...@anarazel.de) wrote:
> On 2015-05-25 13:38:01 -0400, Tom Lane wrote:
> > Andres Freund writes:
> > > On May 24, 2015 7:52:53 AM PDT, Tom Lane wrote:
> > > If we'd merge it with initdb's list I think I'd not be that bad. I'm
> > > thinking of some header declaring it, rough
On 2015-05-25 13:38:01 -0400, Tom Lane wrote:
> Andres Freund writes:
> > On May 24, 2015 7:52:53 AM PDT, Tom Lane wrote:
> > If we'd merge it with initdb's list I think I'd not be that bad. I'm
> > thinking of some header declaring it, roughly like the rmgr list.
>
> pg_log/ is a counterexampl
Andres Freund writes:
> On May 24, 2015 7:52:53 AM PDT, Tom Lane wrote:
>> Christoph Berg writes:
>>> pg_log/ is also admin domain. What about only recursing into
>>> well-known directories + postgresql.auto.conf?
>> The idea that this code would know exactly what's what under $PGDATA
>> scares
On May 24, 2015 7:52:53 AM PDT, Tom Lane wrote:
>Christoph Berg writes:
>> Re: To Andres Freund 2015-05-24 <20150524075244.gb27...@msg.df7cb.de>
>>> Re: Andres Freund 2015-05-24
><20150524005245.gd32...@alap3.anarazel.de>
How about, to avoid masking actual problems, we have a more
diffe
Christoph Berg writes:
> Re: To Andres Freund 2015-05-24 <20150524075244.gb27...@msg.df7cb.de>
>> Re: Andres Freund 2015-05-24 <20150524005245.gd32...@alap3.anarazel.de>
>>> How about, to avoid masking actual problems, we have a more
>>> differentiated logic for the toplevel data directory?
> pg_
Re: To Andres Freund 2015-05-24 <20150524075244.gb27...@msg.df7cb.de>
> Re: Andres Freund 2015-05-24 <20150524005245.gd32...@alap3.anarazel.de>
> > How about, to avoid masking actual problems, we have a more
> > differentiated logic for the toplevel data directory? I think we could
> > just skip al
Re: Andres Freund 2015-05-24 <20150524005245.gd32...@alap3.anarazel.de>
> How about, to avoid masking actual problems, we have a more
> differentiated logic for the toplevel data directory? I think we could
> just skip all non-directory files in there data_directory itself. None
> of the files in t
On 2015-05-23 16:33:29 -0400, Tom Lane wrote:
> Christoph Berg writes:
> > the new fsync-pgdata-on-recovery code tries to open all files using
> > O_RDWR. At least on 9.1, this can make recovery fail:
>
> Hm. I wonder whether it would be all right to just skip files for which
> we get EPERM on o
Re: Tom Lane 2015-05-23 <2284.1432413...@sss.pgh.pa.us>
> Christoph Berg writes:
> > the new fsync-pgdata-on-recovery code tries to open all files using
> > O_RDWR. At least on 9.1, this can make recovery fail:
>
> Hm. I wonder whether it would be all right to just skip files for which
> we get
Christoph Berg writes:
> the new fsync-pgdata-on-recovery code tries to open all files using
> O_RDWR. At least on 9.1, this can make recovery fail:
Hm. I wonder whether it would be all right to just skip files for which
we get EPERM on open(). The argument being that if we can't write to the
f
Hi,
the new fsync-pgdata-on-recovery code tries to open all files using
O_RDWR. At least on 9.1, this can make recovery fail:
* launch postgres, hit ^\ (or otherwise shut down uncleanly)
* touch foo; chmod 444 foo
* launch postgres
LOG: database system was interrupted; last known up at 2015-05-
69 matches
Mail list logo
