Re: pg_combinebackup does not detect missing files

2024-05-18 Thread David Steele
On 5/18/24 21:06, Tomas Vondra wrote: On 5/17/24 14:20, Robert Haas wrote: On Fri, May 17, 2024 at 1:18 AM David Steele wrote: However, I think allowing the user to optionally validate the input would be a good feature. Running pg_verifybackup as a separate step is going to be a more

Re: pg_combinebackup does not detect missing files

2024-05-18 Thread Tomas Vondra
On 5/17/24 14:20, Robert Haas wrote: > On Fri, May 17, 2024 at 1:18 AM David Steele wrote: >> However, I think allowing the user to optionally validate the input >> would be a good feature. Running pg_verifybackup as a separate step is >> going to be a more expensive then verifying/copying at

Re: pg_combinebackup does not detect missing files

2024-05-17 Thread David Steele
On 5/17/24 22:20, Robert Haas wrote: On Fri, May 17, 2024 at 1:18 AM David Steele wrote: However, I think allowing the user to optionally validate the input would be a good feature. Running pg_verifybackup as a separate step is going to be a more expensive then verifying/copying at the same

Re: pg_combinebackup does not detect missing files

2024-05-17 Thread Robert Haas
On Fri, May 17, 2024 at 1:18 AM David Steele wrote: > However, I think allowing the user to optionally validate the input > would be a good feature. Running pg_verifybackup as a separate step is > going to be a more expensive then verifying/copying at the same time. > Even with storage tricks to

Re: pg_combinebackup does not detect missing files

2024-05-16 Thread David Steele
On 4/25/24 00:05, Robert Haas wrote: On Tue, Apr 23, 2024 at 7:23 PM David Steele wrote: I don't understand what you mean here. I thought we were in agreement that verifying contents would cost a lot more. The verification that we can actually do without much cost can only check for missing

Re: pg_combinebackup does not detect missing files

2024-04-24 Thread Robert Haas
On Tue, Apr 23, 2024 at 7:23 PM David Steele wrote: > > I don't understand what you mean here. I thought we were in agreement > > that verifying contents would cost a lot more. The verification that > > we can actually do without much cost can only check for missing files > > in the most recent

Re: pg_combinebackup does not detect missing files

2024-04-23 Thread David Steele
On 4/22/24 23:53, Robert Haas wrote: On Sun, Apr 21, 2024 at 8:47 PM David Steele wrote: I figured that wouldn't be particularly meaningful, and that's pretty much the only kind of validation that's even theoretically possible without a bunch of extra overhead, since we compute checksums on

Re: pg_combinebackup does not detect missing files

2024-04-22 Thread Robert Haas
On Sun, Apr 21, 2024 at 8:47 PM David Steele wrote: > > I figured that wouldn't be particularly meaningful, and > > that's pretty much the only kind of validation that's even > > theoretically possible without a bunch of extra overhead, since we > > compute checksums on entire files rather than,

Re: pg_combinebackup does not detect missing files

2024-04-21 Thread David Steele
On 4/20/24 01:47, Robert Haas wrote: On Thu, Apr 18, 2024 at 7:36 PM David Steele wrote: Sure -- pg_combinebackup would only need to verify the data that it uses. I'm not suggesting that it should do an exhaustive verify of every single backup in the chain. Though I can see how it sounded that

Re: pg_combinebackup does not detect missing files

2024-04-19 Thread Robert Haas
On Thu, Apr 18, 2024 at 7:36 PM David Steele wrote: > Sure -- pg_combinebackup would only need to verify the data that it > uses. I'm not suggesting that it should do an exhaustive verify of every > single backup in the chain. Though I can see how it sounded that way > since with pg_verifybackup

Re: pg_combinebackup does not detect missing files

2024-04-18 Thread David Steele
On 4/19/24 00:50, Robert Haas wrote: On Wed, Apr 17, 2024 at 7:09 PM David Steele wrote: Fair enough. I accept that your reasoning is not random, but I'm still not very satisfied that the user needs to run a separate and rather expensive process to do the verification when pg_combinebackup

Re: pg_combinebackup does not detect missing files

2024-04-18 Thread Robert Haas
On Wed, Apr 17, 2024 at 7:09 PM David Steele wrote: > I think here: > > + pg_basebackup only attempts to verify > > you mean: > > + pg_combinebackup only attempts to verify > > Otherwise this looks good to me. Good catch, thanks. Committed with that change. > Fair enough. I accept that your

Re: pg_combinebackup does not detect missing files

2024-04-17 Thread David Steele
On 4/18/24 01:03, Robert Haas wrote: On Tue, Apr 16, 2024 at 7:25 PM David Steele wrote: But it will not go out of its way to perform checks that are unrelated to its documented purpose. And I don't think it should, especially if we have another tool that already does that. I'm not averse to

Re: pg_combinebackup does not detect missing files

2024-04-17 Thread Robert Haas
On Tue, Apr 16, 2024 at 7:25 PM David Steele wrote: > Except that we are running pg_combinebackup on the incremental, which > the user might reasonably expect to check backup integrity. It actually > does a bunch of integrity checks -- but not this one. I think it's a bad idea to duplicate all

Re: pg_combinebackup does not detect missing files

2024-04-16 Thread David Steele
On 4/16/24 23:50, Robert Haas wrote: On Wed, Apr 10, 2024 at 9:36 PM David Steele wrote: I've been playing around with the incremental backup feature trying to get a sense of how it can be practically used. One of the first things I always try is to delete random files and see what happens.

Re: pg_combinebackup does not detect missing files

2024-04-16 Thread Robert Haas
On Wed, Apr 10, 2024 at 9:36 PM David Steele wrote: > I've been playing around with the incremental backup feature trying to > get a sense of how it can be practically used. One of the first things I > always try is to delete random files and see what happens. > > You can delete pretty much

pg_combinebackup does not detect missing files

2024-04-10 Thread David Steele
Hackers, I've been playing around with the incremental backup feature trying to get a sense of how it can be practically used. One of the first things I always try is to delete random files and see what happens. You can delete pretty much anything you want from the most recent incremental