Re: [BackupPC-users] Converting whole v3 repository to v4?

[Craig Barratt]

There are two options to implement this:

   - truly convert v3 backups to v4 format.  This is quite complex since
   forward deltas (ie, incrementals) would have to be converted into reverse
   deltas; there are lots of cases to consider and testing them all would take
   a lot of time.  The effort and risk (of messing up an old backup) doesn't
   seem worth it.
   - hybrid - convert just the backup storage from V3 to V4, but keep the
   V3 logical dependency (ie: replace the V3 attrib file + hardlinks in each
   directory with just a V4 attrib file).  This is a good bit easier and lower
   risk, as each directory can be updated in a self-contained manner.  The
   testing would need to ensure that V3 viewing / restore etc works with this
   hybrid storage.

If there is enough interest I would consider implementing the hybrid
approach.  But I'm working on rsync-bpc-3.1.2 right now.

Craig

On Tue, Mar 7, 2017 at 6:41 PM, Michael Huntley  wrote:

> Hi Craig -
>
> among the billion other things to do in life - is this reasonably
> possible?  I looked at the perl and came to the considered conclusion, I'm
> just not a perl guy.  However, it would be great to convert older backups
> to v4 - that would reduce my workload on BTRFS snapshops, etc.  No
> hardlinks makes moving large sums of data so much easier..
>
>
> 2cents.
>
> mph
>
> On 2017-03-06 01:58, Pavel Hofman wrote:
>
> Hello,
>
> please is it possible somehow to script the BackupPC_backupDuplicate
> tool to be able to convert not only the last v3 backup to v4, but also
> all previous backups? I would run the script in the background.
>
> The intention is migrating from XFS to ZFS for server consolidation and
> I am afraid I need to get rid of the hardlinks in the 10 yrs of
> historical backups.
>
> I have read about running an arbitrary filesystem over ZFS block device,
> but  I am afraid of poor performance.
>
> Thanks a lot.
>
> Best regards,
>
> Pavel.
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> ___
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
>
>
> 
> --
> Announcing the Oxford Dictionaries API! The API offers world-renowned
> dictionary content that is easy and intuitive to access. Sign up for an
> account today to start using our lexical data to power your apps and
> projects. Get started today and enter our developer competition.
> http://sdm.link/oxford
> ___
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
>
>
--
Announcing the Oxford Dictionaries API! The API offers world-renowned
dictionary content that is easy and intuitive to access. Sign up for an
account today to start using our lexical data to power your apps and
projects. Get started today and enter our developer competition.
http://sdm.link/oxford___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Converting whole v3 repository to v4?

[Michael Huntley]

Hi Craig -  

among the billion other things to do in life - is this reasonably
possible?  I looked at the perl and came to the considered conclusion,
I'm just not a perl guy.  However, it would be great to convert older
backups to v4 - that would reduce my workload on BTRFS snapshops, etc. 
No hardlinks makes moving large sums of data so much easier..

2cents. 

mph 

On 2017-03-06 01:58, Pavel Hofman wrote:

> Hello,
> 
> please is it possible somehow to script the BackupPC_backupDuplicate 
> tool to be able to convert not only the last v3 backup to v4, but also 
> all previous backups? I would run the script in the background.
> 
> The intention is migrating from XFS to ZFS for server consolidation and 
> I am afraid I need to get rid of the hardlinks in the 10 yrs of 
> historical backups.
> 
> I have read about running an arbitrary filesystem over ZFS block device, 
> but  I am afraid of poor performance.
> 
> Thanks a lot.
> 
> Best regards,
> 
> Pavel.
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> ___
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/--
Announcing the Oxford Dictionaries API! The API offers world-renowned
dictionary content that is easy and intuitive to access. Sign up for an
account today to start using our lexical data to power your apps and
projects. Get started today and enter our developer competition.
http://sdm.link/oxford___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] server maintenance: reconstruct missing poolCnt; find/delete references to missing pool files

[Craig Barratt]

It looks like BackupPC_refCountUpdate removes the noPoolCntOk file, even if
it doesn't write any.  So I will fix that (keep the existing file, or
create a new one, if there are no pool files after an fsck).

Craig

On Tue, Mar 7, 2017 at 9:58 AM, Alexander Kobel  wrote:

> Hi Craig,
>
> and thanks a lot for your swift reply.  Comforting to see that both
> messages are known and harmless.
>
> On 2017-03-07 18:24, Craig Barratt wrote:
> >>> BackupPC_refCountUpdate: doing fsck on  #1188 since there are no
> poolCnt files
> >>> BackupPC_refCountUpdate: doing fsck on  #1190 since there are no
> poolCnt files
> >>> ...
> >>> BackupPC_refCountUpdate: host  got 0 errors (took 5 secs)
> >>
> >> The backups in question seem to be fully intact; [...]
> >>
> > This is perfectly ok.  BackupPC 4.0.0alpha3 and prior 4.x versions
> > didn't store reference counts per backup.  [...]  So
> BackupPC_refCountUpdate is
> > simply adding reference counts to backups done by BackupPC
> > 4.0.0alpha3.  It's a one-time thing.
>
> Got that, and...
>
> > There might be an issue that an incremental done by BackupPC
> > 4.0.0alpha3 with no changes will have an empty backup tree, and
> > BackupPC_refCntUpdate will continually report that there are no
> > poolCnt files for that backup.  That's benign.  [...]
>
> ... indeed that's the case here.  It's backups of a data directory on a
> server that rarely changes.
>
> > In 4.0.0, BackupPC_dump flags that by creating a file
> > "HOST/NNN/refCnt/noPoolCntOk, which makes BackupPC_refCntUpdate
> > quietly ignore that backup.  Perhaps I should have
> > BackupPC_refCntUpdate notice that legacy case and create the
> > noPoolCntOk file...
>
> Certainly low priority, but you might keep it on the list if it's not a
> lot of work.
> On the other hand, now everyone who searches for the warning will find
> this mailing list post, and be pleased to hear that a simple
>   touch //refCnt/noPoolCntOk
> gets rid of the warning.  (Sanity check before: confirm that //
> did not contain anything but backupInfo and an empty refCnt/ directory.)
>
> >> BackupPC_refCountUpdate: missing pool file
>  count 30
> >> BackupPC_refCountUpdate: missing pool file
> 0601e1b90a7f92ce4cffa588ef2cc9da count 1
> >> ...
> >> BackupPC_refCountUpdate: missing pool file
> ea1bd7ab2e00 count 1
> >
> > This is a bug in rsync-bpc [...] (yes, I had "<" instead of "<="...
> doh!).
>
> Oh, well...  He that has never wrote that error, let him first cast a
> stone... ;-)
>
> > Future backups with 4.0.0 (assuming the same file exists on the
> > client) will be updated with the correct digest, but the old backups
> > will still have the wrong one.  The errors will go away when the
> > corresponding backups eventually expire.
>
> Okay.  Just important to know that it will "fix itself" rather than
> getting worse over time.
>
>
> Thanks again for your reply, and thanks for such a great overall program!
>
>
> Alexander
>
>
> > On Tue, Mar 7, 2017 at 6:56 AM, Alexander Kobel  > > wrote:
> >
> > Dear all,
> >
> > I have a rather small, private, non-$$$M-mission-critical instance of
> > a BackupPC server running for years (that went from several
> > 3.something through 4.0.0alpha3 to, recently, 4.0.0). After the 4.0.0
> > migration, I decided to run again some semi-manual maintenance (read:
> > fsck and refCountUpdate). Yet, after an (admitted, more or less
> > random) sequence of BackupPC_fsck BackupPC_fsck -f BackupPC_fsck -f
> > -s BackupPC_refCountUpdate -m BackupPC_refCountUpdate -m -F -c
> > BackupPC_refCountUpdate -m -F -c -s BackupPC_fixupBackupSummary
> > BackupPC_nightly 0 255 I'm still stuck with the following messages:
> >
> >> BackupPC_refCountUpdate: doing fsck on  #1188 since there are
> >> no poolCnt files BackupPC_refCountUpdate: doing fsck on 
> >> #1190 since there are no poolCnt files ... BackupPC_refCountUpdate:
> >> host  got 0 errors (took 5 secs)
> >
> > The backups in question seem to be fully intact; some are full
> > backups, some are incremental. It's just on a minority of backups
> > (appx. 15 out of 350 backups), and fortunately on small ones where
> > fsck does not take ages, so it does not bother me too much.
> > Nevertheless, can the missing poolCnt data be recomputed? fsck seems
> > to do the counting from scratch; can this be stored?
> >
> >> BackupPC_fsck: building main count database
> >> BackupPC_refCountUpdate: missing pool file
> >>  count 30 BackupPC_refCountUpdate:
> >> missing pool file 0601e1b90a7f92ce4cffa588ef2cc9da count 1 ...
> >> BackupPC_refCountUpdate: missing pool file
> >> ea1bd7ab2e00 count 1 ...
> >> BackupPC_refCountUpdate total errors: 70 BackupPC_fsck: Calling
> >> poolCountUpdate
> >
> > IIUC, this means that there are reference to files with that hash
> > *somewhere* in the backups, but the respective files are missing 

Re: [BackupPC-users] server maintenance: reconstruct missing poolCnt; find/delete references to missing pool files

[Alexander Kobel]

Hi Craig,

and thanks a lot for your swift reply.  Comforting to see that both messages 
are known and harmless.

On 2017-03-07 18:24, Craig Barratt wrote:
>>> BackupPC_refCountUpdate: doing fsck on  #1188 since there are no 
>>> poolCnt files
>>> BackupPC_refCountUpdate: doing fsck on  #1190 since there are no 
>>> poolCnt files
>>> ...
>>> BackupPC_refCountUpdate: host  got 0 errors (took 5 secs)
>>
>> The backups in question seem to be fully intact; [...]
>> 
> This is perfectly ok.  BackupPC 4.0.0alpha3 and prior 4.x versions
> didn't store reference counts per backup.  [...]  So BackupPC_refCountUpdate 
> is
> simply adding reference counts to backups done by BackupPC
> 4.0.0alpha3.  It's a one-time thing.

Got that, and...

> There might be an issue that an incremental done by BackupPC
> 4.0.0alpha3 with no changes will have an empty backup tree, and
> BackupPC_refCntUpdate will continually report that there are no
> poolCnt files for that backup.  That's benign.  [...]

... indeed that's the case here.  It's backups of a data directory on a server 
that rarely changes.

> In 4.0.0, BackupPC_dump flags that by creating a file
> "HOST/NNN/refCnt/noPoolCntOk, which makes BackupPC_refCntUpdate
> quietly ignore that backup.  Perhaps I should have
> BackupPC_refCntUpdate notice that legacy case and create the
> noPoolCntOk file...

Certainly low priority, but you might keep it on the list if it's not a lot of 
work.
On the other hand, now everyone who searches for the warning will find this 
mailing list post, and be pleased to hear that a simple
  touch //refCnt/noPoolCntOk
gets rid of the warning.  (Sanity check before: confirm that // did 
not contain anything but backupInfo and an empty refCnt/ directory.)

>> BackupPC_refCountUpdate: missing pool file  
>> count 30
>> BackupPC_refCountUpdate: missing pool file 0601e1b90a7f92ce4cffa588ef2cc9da 
>> count 1
>> ...
>> BackupPC_refCountUpdate: missing pool file ea1bd7ab2e00 
>> count 1
> 
> This is a bug in rsync-bpc [...] (yes, I had "<" instead of "<="... doh!).

Oh, well...  He that has never wrote that error, let him first cast a stone... 
;-)

> Future backups with 4.0.0 (assuming the same file exists on the
> client) will be updated with the correct digest, but the old backups
> will still have the wrong one.  The errors will go away when the
> corresponding backups eventually expire.

Okay.  Just important to know that it will "fix itself" rather than getting 
worse over time.


Thanks again for your reply, and thanks for such a great overall program!


Alexander


> On Tue, Mar 7, 2017 at 6:56 AM, Alexander Kobel  > wrote:
> 
> Dear all,
> 
> I have a rather small, private, non-$$$M-mission-critical instance of
> a BackupPC server running for years (that went from several
> 3.something through 4.0.0alpha3 to, recently, 4.0.0). After the 4.0.0
> migration, I decided to run again some semi-manual maintenance (read:
> fsck and refCountUpdate). Yet, after an (admitted, more or less
> random) sequence of BackupPC_fsck BackupPC_fsck -f BackupPC_fsck -f
> -s BackupPC_refCountUpdate -m BackupPC_refCountUpdate -m -F -c 
> BackupPC_refCountUpdate -m -F -c -s BackupPC_fixupBackupSummary 
> BackupPC_nightly 0 255 I'm still stuck with the following messages:
> 
>> BackupPC_refCountUpdate: doing fsck on  #1188 since there are
>> no poolCnt files BackupPC_refCountUpdate: doing fsck on 
>> #1190 since there are no poolCnt files ... BackupPC_refCountUpdate:
>> host  got 0 errors (took 5 secs)
> 
> The backups in question seem to be fully intact; some are full
> backups, some are incremental. It's just on a minority of backups
> (appx. 15 out of 350 backups), and fortunately on small ones where
> fsck does not take ages, so it does not bother me too much.
> Nevertheless, can the missing poolCnt data be recomputed? fsck seems
> to do the counting from scratch; can this be stored?
> 
>> BackupPC_fsck: building main count database 
>> BackupPC_refCountUpdate: missing pool file
>>  count 30 BackupPC_refCountUpdate:
>> missing pool file 0601e1b90a7f92ce4cffa588ef2cc9da count 1 ... 
>> BackupPC_refCountUpdate: missing pool file
>> ea1bd7ab2e00 count 1 ... 
>> BackupPC_refCountUpdate total errors: 70 BackupPC_fsck: Calling
>> poolCountUpdate
> 
> IIUC, this means that there are reference to files with that hash
> *somewhere* in the backups, but the respective files are missing from
> the cpool. Since the number is very low, I'm not really worried; in
> particular, I'm pretty sure that the pool file with hash 0*32 is a
> spurious one. Nevertheless, I'd like to see which filenames/entries
> refer to the missing pool files, and if possible delete those
> altogether to get a "clean" state without nagging messages.
> Unfortunately, I'm entirely clueless on how to approach that.
> 
> FWIW: the server runs on CentOS 7.3, on an 

Re: [BackupPC-users] server maintenance: reconstruct missing poolCnt; find/delete references to missing pool files

[Craig Barratt]

>
> > BackupPC_refCountUpdate: doing fsck on  #1188 since there are no
> poolCnt files
> > BackupPC_refCountUpdate: doing fsck on  #1190 since there are no
> poolCnt files
> > ...
> > BackupPC_refCountUpdate: host  got 0 errors (took 5 secs)
> The backups in question seem to be fully intact; some are full backups,
> some are incremental. It's just on a minority of backups (appx. 15 out of
> 350 backups), and fortunately on small ones where fsck does not take ages,
> so it does not bother me too much. Nevertheless, can the missing poolCnt
> data be recomputed? fsck seems to do the counting from scratch; can this be
> stored?


This is perfectly ok.  BackupPC 4.0.0alpha3 and prior 4.x versions didn't
store reference counts per backup.  Only the reference counts for the
entire host were maintained (in addition to the pool totals for all
hosts).  In 4.0.0, I changed that so reference counts were also stored
per-backup (which makes it easier to delete backups and to recompute the
per-host ref counts).  So BackupPC_refCountUpdate is simply adding
reference counts to backups done by BackupPC 4.0.0alpha3.  It's a one-time
thing.

There might be an issue that an incremental done by BackupPC 4.0.0alpha3
with no changes will have an empty backup tree, and BackupPC_refCntUpdate
will continually report that there are no poolCnt files for that backup.
That's benign.  In 4.0.0, BackupPC_dump flags that by creating a file
"HOST/NNN/refCnt/noPoolCntOk, which makes BackupPC_refCntUpdate quietly
ignore that backup.  Perhaps I should have BackupPC_refCntUpdate notice
that legacy case and create the noPoolCntOk file...

> BackupPC_refCountUpdate: missing pool file
>  count 30
> > BackupPC_refCountUpdate: missing pool file
> 0601e1b90a7f92ce4cffa588ef2cc9da count 1
> > ...
> > BackupPC_refCountUpdate: missing pool file
> ea1bd7ab2e00 count 1


This is a bug in rsync-bpc (and BackupPC::XS) that was fixed a couple of
weeks ago.  It happened about 2% of the time the attrib file for a large
directory was written (attrib file sizes >256k, approx 5k files depending
on file name lengths).  If the md5 digest of the last file written to the
256k staging buffer exactly lined up with the end of the buffer, the digest
for that file wasn't written correctly (yes, I had "<" instead of "<="...
doh!).

Future backups with 4.0.0 (assuming the same file exists on the client)
will be updated with the correct digest, but the old backups will still
have the wrong one.  The errors will go away when the corresponding backups
eventually expire.

Craig

On Tue, Mar 7, 2017 at 6:56 AM, Alexander Kobel  wrote:

> Dear all,
>
> I have a rather small, private, non-$$$M-mission-critical instance of a
> BackupPC server running for years (that went from several 3.something
> through 4.0.0alpha3 to, recently, 4.0.0).
> After the 4.0.0 migration, I decided to run again some semi-manual
> maintenance (read: fsck and refCountUpdate). Yet, after an (admitted, more
> or less random) sequence of
>   BackupPC_fsck
>   BackupPC_fsck -f
>   BackupPC_fsck -f -s
>   BackupPC_refCountUpdate -m
>   BackupPC_refCountUpdate -m -F -c
>   BackupPC_refCountUpdate -m -F -c -s
>   BackupPC_fixupBackupSummary
>   BackupPC_nightly 0 255
> I'm still stuck with the following messages:
>
> > BackupPC_refCountUpdate: doing fsck on  #1188 since there are no
> poolCnt files
> > BackupPC_refCountUpdate: doing fsck on  #1190 since there are no
> poolCnt files
> > ...
> > BackupPC_refCountUpdate: host  got 0 errors (took 5 secs)
>
> The backups in question seem to be fully intact; some are full backups,
> some are incremental. It's just on a minority of backups (appx. 15 out of
> 350 backups), and fortunately on small ones where fsck does not take ages,
> so it does not bother me too much. Nevertheless, can the missing poolCnt
> data be recomputed? fsck seems to do the counting from scratch; can this be
> stored?
>
> > BackupPC_fsck: building main count database
> > BackupPC_refCountUpdate: missing pool file 
> count 30
> > BackupPC_refCountUpdate: missing pool file 0601e1b90a7f92ce4cffa588ef2cc9da
> count 1
> > ...
> > BackupPC_refCountUpdate: missing pool file ea1bd7ab2e00
> count 1
> > ...
> > BackupPC_refCountUpdate total errors: 70
> > BackupPC_fsck: Calling poolCountUpdate
>
> IIUC, this means that there are reference to files with that hash
> *somewhere* in the backups, but the respective files are missing from the
> cpool. Since the number is very low, I'm not really worried; in particular,
> I'm pretty sure that the pool file with hash 0*32 is a spurious one.
> Nevertheless, I'd like to see which filenames/entries refer to the missing
> pool files, and if possible delete those altogether to get a "clean" state
> without nagging messages. Unfortunately, I'm entirely clueless on how to
> approach that.
>
> FWIW: the server runs on CentOS 7.3, on an ext3