Re: [HACKERS] pg_basebackup: could not get transaction log end position from server: FATAL: could not open file "./pg_hba.conf~": Permission denied
On 05/16/2014 09:20 AM, Magnus Hagander wrote: On Fri, May 16, 2014 at 5:46 PM, Joshua D. Drake mailto:j...@commandprompt.com>> wrote: At a minimum: Check to see if there is going to be a permission error BEFORE the base backup begins: starting basebackup: checking perms: ERROR no access to pg_hba.conf~ base backup will fail That's pretty much what it does if you enable progress meter. I realize you don't necessarily want that one, but we could have a switch that still tells the server to measure the size, but not actually print the output? While it costs a bit of overhead to do that, that's certainly something that's a lot more safe than ignoring errors. That seems reasonable. JD -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_basebackup: could not get transaction log end position from server: FATAL: could not open file "./pg_hba.conf~": Permission denied
On 2014-05-16 18:29:25 +0200, Magnus Hagander wrote: > On Fri, May 16, 2014 at 6:25 PM, Andres Freund wrote: > > > On 2014-05-16 18:20:35 +0200, Magnus Hagander wrote: > > > On Fri, May 16, 2014 at 5:46 PM, Joshua D. Drake > >wrote: > > > > > > > At a minimum: > > > > > > > > Check to see if there is going to be a permission error BEFORE the base > > > > backup begins: > > > > > > > > starting basebackup: > > > > checking perms: ERROR no access to pg_hba.conf~ base backup will fail > > > > > > > > > That's pretty much what it does if you enable progress meter. I realize > > you > > > don't necessarily want that one, but we could have a switch that still > > > tells the server to measure the size, but not actually print the output? > > > While it costs a bit of overhead to do that, that's certainly something > > > that's a lot more safe than ignoring errors. > > > > Don't think it'll show you that error - that mode only stats() files, > > right? So you'd need to add access() or open()s. > > > > > You're right, we don't. I thought we did, but was clearly remembering wrong. > > I guess we could add an access() call to that codepath though. Not sure if > that's going to cause any real overhead compared to the rest of what we're > doing anyway? It's not free. But I don't think it'd seriously matter in comparison. But it doesn't protect you if the file is created during the backup - which as you know can take a long time. For example because somebody felt the need to increase wal_keep_segments. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_basebackup: could not get transaction log end position from server: FATAL: could not open file "./pg_hba.conf~": Permission denied
On Fri, May 16, 2014 at 6:25 PM, Andres Freund wrote: > On 2014-05-16 18:20:35 +0200, Magnus Hagander wrote: > > On Fri, May 16, 2014 at 5:46 PM, Joshua D. Drake >wrote: > > > > > At a minimum: > > > > > > Check to see if there is going to be a permission error BEFORE the base > > > backup begins: > > > > > > starting basebackup: > > > checking perms: ERROR no access to pg_hba.conf~ base backup will fail > > > > > > That's pretty much what it does if you enable progress meter. I realize > you > > don't necessarily want that one, but we could have a switch that still > > tells the server to measure the size, but not actually print the output? > > While it costs a bit of overhead to do that, that's certainly something > > that's a lot more safe than ignoring errors. > > Don't think it'll show you that error - that mode only stats() files, > right? So you'd need to add access() or open()s. > > You're right, we don't. I thought we did, but was clearly remembering wrong. I guess we could add an access() call to that codepath though. Not sure if that's going to cause any real overhead compared to the rest of what we're doing anyway? -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/
Re: [HACKERS] pg_basebackup: could not get transaction log end position from server: FATAL: could not open file "./pg_hba.conf~": Permission denied
On 2014-05-16 18:20:35 +0200, Magnus Hagander wrote: > On Fri, May 16, 2014 at 5:46 PM, Joshua D. Drake > wrote: > > > At a minimum: > > > > Check to see if there is going to be a permission error BEFORE the base > > backup begins: > > > > starting basebackup: > > checking perms: ERROR no access to pg_hba.conf~ base backup will fail > > > That's pretty much what it does if you enable progress meter. I realize you > don't necessarily want that one, but we could have a switch that still > tells the server to measure the size, but not actually print the output? > While it costs a bit of overhead to do that, that's certainly something > that's a lot more safe than ignoring errors. Don't think it'll show you that error - that mode only stats() files, right? So you'd need to add access() or open()s. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_basebackup: could not get transaction log end position from server: FATAL: could not open file "./pg_hba.conf~": Permission denied
On Fri, May 16, 2014 at 5:46 PM, Joshua D. Drake wrote: > At a minimum: > > Check to see if there is going to be a permission error BEFORE the base > backup begins: > > starting basebackup: > checking perms: ERROR no access to pg_hba.conf~ base backup will fail That's pretty much what it does if you enable progress meter. I realize you don't necessarily want that one, but we could have a switch that still tells the server to measure the size, but not actually print the output? While it costs a bit of overhead to do that, that's certainly something that's a lot more safe than ignoring errors. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/
Re: [HACKERS] pg_basebackup: could not get transaction log end position from server: FATAL: could not open file "./pg_hba.conf~": Permission denied
On 05/16/2014 08:48 AM, Andres Freund wrote: Hi, On 2014-05-16 08:45:12 -0700, Joshua D. Drake wrote: Software is supposed to make our lives easier, not harder. I should be able to evaluate the errors for the conditions they create. This is why rsync is and for the forseeable future will be king for creating base backups. It's dangerous to ignore errors rsync errors other than 'file vanished'. This hardly is an argument for your position. Are you reading what I write? I said. "I should be able to evaluate the errors for the conditions they create." I never suggested ignoring anything. The point is RSYNC gives me a chance at success, pg_basebackup does not (in respect to this specific condition). JD -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_basebackup: could not get transaction log end position from server: FATAL: could not open file "./pg_hba.conf~": Permission denied
Hi, On 2014-05-16 08:45:12 -0700, Joshua D. Drake wrote: > Software is supposed to make our lives easier, not harder. I should be able > to evaluate the errors for the conditions they create. This is why rsync is > and for the forseeable future will be king for creating base backups. It's dangerous to ignore errors rsync errors other than 'file vanished'. This hardly is an argument for your position. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_basebackup: could not get transaction log end position from server: FATAL: could not open file "./pg_hba.conf~": Permission denied
At a minimum: Check to see if there is going to be a permission error BEFORE the base backup begins: starting basebackup: checking perms: ERROR no access to pg_hba.conf~ base backup will fail JD -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_basebackup: could not get transaction log end position from server: FATAL: could not open file "./pg_hba.conf~": Permission denied
On 05/16/2014 08:19 AM, Tom Lane wrote: pg_hba.conf~ is not an important file. Rather than blaming the messenger, you should be asking why there are files in $PGDATA that the server can't read. That's a recipe for trouble no matter what. Or in words of one syllable: this is a bug in your editor, not in Postgres. Hardly and shows a distinct lack of user space experience. It also shows how useless pg_basebackup "can" be. Basically you are saying, "Well yeah, there is this rogue file that doesn't belong, fark it... we will blow away a 2TB base backup and make you start over because.. meh, pg_basebackup is lazy." Software is supposed to make our lives easier, not harder. I should be able to evaluate the errors for the conditions they create. This is why rsync is and for the forseeable future will be king for creating base backups. JD regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_basebackup: could not get transaction log end position from server: FATAL: could not open file "./pg_hba.conf~": Permission denied
On 2014-05-16 08:13:04 -0700, Joshua D. Drake wrote: > On 05/16/2014 07:30 AM, Andres Freund wrote: > > > >Hi, > > > >On 2014-05-16 07:28:42 -0700, Joshua D. Drake wrote: > >>Can we get that fixed please? It seems rather bad behavior for pg_basebackup > >>to fatal out because of the permissions on a backup file of all things. > >>Instead, we should do WARNING and say skipped. > > > >Doesn't sound like a good idea to me. We'd need to have a catalog of > >common unimportant fileendings and such. We surely *do* want to error > >out when we fail to copy an important file.they > > > > pg_hba.conf~ is not an important file. Where do we know that from? > We know what files are important, especially in $PGDATA, they aren't > variable, so why is pg_basebackup failing on a file it should know or care > nothing about? No, we don't necessarily. It'd e.g. bad to succeed if postgresql.conf includes another file and we fail when backing that up even though it's in the data directory. But even otherwise it'd be a non-neglegible amount of code to enumerate possibly important files (which wouldn't fully reliable. We can't access the catalogs). Code that's only there to work around a user doing something bad that's trivially fixable. Nah. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_basebackup: could not get transaction log end position from server: FATAL: could not open file "./pg_hba.conf~": Permission denied
"Joshua D. Drake" writes: > On 05/16/2014 07:30 AM, Andres Freund wrote: >> On 2014-05-16 07:28:42 -0700, Joshua D. Drake wrote: >>> Can we get that fixed please? It seems rather bad behavior for pg_basebackup >>> to fatal out because of the permissions on a backup file of all things. >>> Instead, we should do WARNING and say skipped. >> Doesn't sound like a good idea to me. We'd need to have a catalog of >> common unimportant fileendings and such. We surely *do* want to error >> out when we fail to copy an important file.they > pg_hba.conf~ is not an important file. Rather than blaming the messenger, you should be asking why there are files in $PGDATA that the server can't read. That's a recipe for trouble no matter what. Or in words of one syllable: this is a bug in your editor, not in Postgres. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_basebackup: could not get transaction log end position from server: FATAL: could not open file "./pg_hba.conf~": Permission denied
On 05/16/2014 07:30 AM, Andres Freund wrote: Hi, On 2014-05-16 07:28:42 -0700, Joshua D. Drake wrote: Can we get that fixed please? It seems rather bad behavior for pg_basebackup to fatal out because of the permissions on a backup file of all things. Instead, we should do WARNING and say skipped. Doesn't sound like a good idea to me. We'd need to have a catalog of common unimportant fileendings and such. We surely *do* want to error out when we fail to copy an important file.they pg_hba.conf~ is not an important file. We know what files are important, especially in $PGDATA, they aren't variable, so why is pg_basebackup failing on a file it should know or care nothing about? JD -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_basebackup: could not get transaction log end position from server: FATAL: could not open file "./pg_hba.conf~": Permission denied
Hi, On 2014-05-16 07:28:42 -0700, Joshua D. Drake wrote: > Can we get that fixed please? It seems rather bad behavior for pg_basebackup > to fatal out because of the permissions on a backup file of all things. > Instead, we should do WARNING and say skipped. Doesn't sound like a good idea to me. We'd need to have a catalog of common unimportant fileendings and such. We surely *do* want to error out when we fail to copy an important file. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
