[BackupPC-users] Disk space used far higher than reported pool size

[Craig O'Brien]

On the General Server Information page, it says "Pool is 2922.42GB
comprising 6061942 files and 4369 directories," but our pool file system
which contains nothing but backuppc and is 11 TB in size is 100% full.

I'm confused how this happened and even ran the BackupPC_nightly script by
hand which didn't seem to clear up any space. Judging by the reported pool
size it should be less than 30% full. I could really use some help. Thanks
in advance for any ideas on how to go about troubleshooting this.

Regards,
Craig
--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Marcel Meckel]

Hi,

> On the General Server Information page, it says "Pool is 2922.42GB
> comprising 6061942 files and 4369 directories," but our pool file system
> which contains nothing but backuppc and is 11 TB in size is 100% full.

some details would be great!

It's a bit hard to guess your setup details...

Which type of filesystem? xfs/ext4/...

Output of "df"

Output of "df -i"

Mount options from /etc/fstab

Regards

Marcel

-- 
Registrierter Linux User #307343

--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Craig O'Brien]

File Type is ext4.

bash-4.1$ df
Filesystem   1K-blocks  Used Available Use% Mounted on
/dev/mapper/vg_harp-lv_root
  51615740  10132616  40958900  20% /
tmpfs  4019640   828   4018812   1% /dev/shm
/dev/md127p1495844135596334648  29% /boot
/dev/mapper/vg_harp-lv_home
 178137608191984 168896744   1% /home
/dev/sda110985539464 10370739976  28862288 100% /backup
naslite:/export/Disk-1
 961424128 67456 339196672  65% /mnt/naslite


bash-4.1$ df -i
FilesystemInodes   IUsed   IFree IUse% Mounted on
/dev/mapper/vg_harp-lv_root
 3238400  171134 30672666% /
tmpfs1004910   4 10049061% /dev/shm
/dev/md127p1  128016  66  1279501% /boot
/dev/mapper/vg_harp-lv_home
 11313152  28 113131241% /home
/dev/sda12929688576 20084766 29096038101% /backup
naslite:/export/Disk-1
 122109952  195254 1219146981% /mnt/naslite
bash-4.1$


bash-4.1$ cat /etc/fstab

#
# /etc/fstab
# Created by anaconda on Fri Dec 16 12:02:05 2011
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/vg_harp-lv_root /   ext4defaults
 1 1
UUID=705ad8c5-3a17-4aac-aab2-e2fc1ce010c7 /boot   ext4
 defaults1 2
/dev/mapper/vg_harp-lv_home /home   ext4defaults
 1 2
/dev/mapper/vg_harp-lv_swap swapswapdefaults
 0 0
UUID=a76d2bf2-3649-4941-9db3-6129aa45d873 swapswap
 defaults0 0
tmpfs   /dev/shmtmpfs   defaults0 0
devpts  /dev/ptsdevpts  gid=5,mode=620  0 0
sysfs   /syssysfs   defaults0 0
proc/proc   procdefaults0 0
/dev/sda1   /backup ext4defaults0 4
naslite:/export/Disk-1  /mnt/naslitenfs rw,hard,intr0   0
bash-4.1$

Backuppc is at version 3.2.1. OS is CentOS release 6.4


Regards,
Craig


On Tue, Oct 29, 2013 at 2:35 PM, Marcel Meckel <
mailinglist+backuppc-us...@foobar0815.de> wrote:

> Hi,
>
> > On the General Server Information page, it says "Pool is 2922.42GB
> > comprising 6061942 files and 4369 directories," but our pool file system
> > which contains nothing but backuppc and is 11 TB in size is 100% full.
>
> some details would be great!
>
> It's a bit hard to guess your setup details...
>
> Which type of filesystem? xfs/ext4/...
>
> Output of "df"
>
> Output of "df -i"
>
> Mount options from /etc/fstab
>
> Regards
>
> Marcel
>
> --
> Registrierter Linux User #307343
>
>
> --
> Android is increasing in popularity, but the open development platform that
> developers love is also attractive to malware creators. Download this white
> paper to learn more about secure code signing practices that can help keep
> Android apps secure.
> http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
> ___
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
>
--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Timothy J Massey]

"Craig O'Brien"  wrote on 10/29/2013 01:53:31 PM:

> On the General Server Information page, it says "Pool is 2922.42GB 
> comprising 6061942 files and 4369 directories," but our pool file 
> system which contains nothing but backuppc and is 11 TB in size is 100% 
full.

My strong guess is that, while you *think* nothing else is out there, that 
is not the case!  :)

> I'm confused how this happened and even ran the BackupPC_nightly 
> script by hand which didn't seem to clear up any space. Judging by 
> the reported pool size it should be less than 30% full. I could 
> really use some help. Thanks in advance for any ideas on how to go 
> about troubleshooting this.

>From the other message, it seems that the filesystem you're worried about 
is /home.  What is the TopDir of BackupPC?  I assume it's something like 
/home/backuppc (or I sure hope it is!).  Go to that path and type:

du -hs

This will take a *long* time:  BackupPC has a *lot* of files.  I would 
also hope that you see a number very similar to the pool size report 
above.

So, if you find that your BackupPC TopDir contents (what you verified with 
du -hs) reports match the GUI, then you know that there's something on the 
drive but *outside* of the BackupPC TopDir.  Find it and delete it.

Because du -hs of the pool takes so long, you could, of course, do it the 
*other* way:  do a du -hs of each directory *besides* your TopDir and see 
how much space is being used by them.  Depends on how many other folders 
you have how hard that would be.  But when you see some other folder using 
8TB, you'll know where the space went!

However, if you find that your du -hs does *not* match your GUI report, 
then you have to look more closely.  Have you *EVER* done anything from 
the command line inside of the TopDir?  Given that you mention running 
BackupPC_nightly by hand, I suspect you of monkeying with things, and that 
very well may have broken things.  Tell us what you did!  :)

(It's not that you can't run BackupPC_nightly by hand;  you can.  It's 
more that if you're brave enough to run it by hand, there's no telling 
what *else* you might have done, and how you might have broken things!  :) 
 ).


If you can't tell, I suspect something outside of BackupPC has used the 
space, *or* that you moved/copied/etc. something using tools outside of 
the BackupPC system and have broken things unintentionally.  It is very 
unlikely that BackupPC is wrong on its pool report.  Most likely it's 
something *else* that is consuming the space, and because of that all the 
BackupPC_nightly in the world isn't going to free up the space.

Tim Massey


 
Out of the Box Solutions, Inc. 
Creative IT Solutions Made Simple!
http://www.OutOfTheBoxSolutions.com
tmas...@obscorp.com 
 
22108 Harper Ave.
St. Clair Shores, MI 48080
Office: (800)750-4OBS (4627)
Cell: (586)945-8796 
--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Craig O'Brien]

The topdir is /var/lib/BackupPC which is a link to /backup

If I do an   ls -l /var/lib
I get a bunch of other directories as well as:
lrwxrwxrwx.  1 rootroot   7 Dec 17  2011 BackupPC -> /backup

bash-4.1$ ls -l /backup
total 20
drwxr-x---. 18 backuppc root 4096 Oct 25 21:01 cpool
drwx--.  2 root root 4096 Dec 17  2011 lost+found
drwxr-x---. 76 backuppc root 4096 Oct 24 16:00 pc
drwxr-x---.  2 backuppc root 4096 Dec 24  2012 pool
drwxr-x---.  2 backuppc root 4096 Oct 29 01:05 trash
bash-4.1$

It's only backuppc stuff on there. I did it this way to give the backuppc
pool a really large drive to itself. As far as things done from the command
line I've deleted computers inside of the pc directory that I no longer
needed to backup. From my understanding that combined with removing the pc
from the /etc/BackupPC/hosts file would free up any space those backups
used to use in the pool. I've manually stopped and started the backuppc
daemon when I've made config changes, or added/removed a pc. At one point I
had almost all of the pc's being backed up with SMB, and switched them all
to using rsync.

I ran the du -hs command you recommended, I'll post the results when it
eventually finishes. Thank you.
​
--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Marcel Meckel]

> I've deleted computers inside of the pc directory that I no longer
> needed to backup. From my understanding that combined with removing the pc
> from the /etc/BackupPC/hosts file would free up any space those backups
> used to use in the pool.

No.

You'll have to wait until the next BackupPC_nightly jobs ran through
the pool completely.

If you only traverse half the pool each night it takes 2 days (and so
on).

Regards
Marcel

-- 
Registrierter Linux User #307343

--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[backuppc]

Craig O'Brien wrote at about 13:53:31 -0400 on Tuesday, October 29, 2013:
 > On the General Server Information page, it says "Pool is 2922.42GB
 > comprising 6061942 files and 4369 directories," but our pool file system
 > which contains nothing but backuppc and is 11 TB in size is 100% full.
 > 
 > I'm confused how this happened and even ran the BackupPC_nightly script by
 > hand which didn't seem to clear up any space. Judging by the reported pool
 > size it should be less than 30% full. I could really use some help. Thanks
 > in advance for any ideas on how to go about troubleshooting this.
 > 
 > Regards,
 > Craig
 > ---

I suspect that you have (multiple) backups in the pc tree that are not
linked to the pool...

The following is a quick-and-dirty hack to find non-zero length
*files* in the pc tree that have fewer than 2 hard links (note this
will miss files that are linked to themselves but not to the pool):

cd /var/lib/BackupPC/pc/; find */*/* -type f -links -2 -size +0 | grep -v 
"^/[^/]*/[0-9]*/backupInfo"

--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Timothy J Massey]

"Craig O'Brien"  wrote on 10/29/2013 03:30:46 PM:

> The topdir is /var/lib/BackupPC which is a link to /backup

I missed that in your previous e-mail.  Stupid proportional fonts...

(And you might want add a -h for commands like du and df:  the -h is for 
human-readable...  When the numbers are for things in the *terabytes*, 
it's a lot of digits to manage...)

> If I do an   ls -l /var/lib
> I get a bunch of other directories as well as:
> lrwxrwxrwx.  1 rootroot   7 Dec 17  2011 BackupPC -> /backup
> 
> bash-4.1$ ls -l /backup
> total 20
> drwxr-x---. 18 backuppc root 4096 Oct 25 21:01 cpool
> drwx--.  2 root root 4096 Dec 17  2011 lost+found
> drwxr-x---. 76 backuppc root 4096 Oct 24 16:00 pc
> drwxr-x---.  2 backuppc root 4096 Dec 24  2012 pool
> drwxr-x---.  2 backuppc root 4096 Oct 29 01:05 trash
> bash-4.1$

That is much clearer, thank you.

> It's only backuppc stuff on there. I did it this way to give the 
> backuppc pool a really large drive to itself.

Sounds reasonable.

> As far as things done 
> from the command line I've deleted computers inside of the pc 
> directory that I no longer needed to backup. From my understanding 
> that combined with removing the pc from the /etc/BackupPC/hosts file
> would free up any space those backups used to use in the pool.

The way the pooling works is that any files with only one hardlink are 
deleted from the pool.  Given that BackupPC is saying that the pool is 
only <3TB big, then your problem is *not* that there are things in the 
pool that aren't anywhere else.  Your problem is the exact opposite:  you 
have files somewhere else that are *not* part of the pool!

> I've 
> manually stopped and started the backuppc daemon when I've made 
> config changes, or added/removed a pc. At one point I had almost all
> of the pc's being backed up with SMB, and switched them all to using 
rsync.

Again, I could imagine lots of ways this might explode your pool, but you 
have the exact opposite problem:  your pool is too small!

Please run Jeff's command to find out where you have files that are not in 
the pool.  That will be most informative.

> I ran the du -hs command you recommended, I'll post the results when
> it eventually finishes. Thank you.

I doubt that will help if you did it from /backup.  The point of that was 
to isolate other non-BackupPC folders.

Check lost+found and trash while you're at it and see what's in there. 
They should both be empty.

I'm with Jeff:  I think that you have multiple PC trees that are not part 
of the pool.  How you managed that I'm not sure.  But you need to find 
those files and clean them up.  Start with Jeff's command and go from 
there.

Tim Massey


 
Out of the Box Solutions, Inc. 
Creative IT Solutions Made Simple!
http://www.OutOfTheBoxSolutions.com
tmas...@obscorp.com 
 
22108 Harper Ave.
St. Clair Shores, MI 48080
Office: (800)750-4OBS (4627)
Cell: (586)945-8796 
--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Les Mikesell]

On Tue, Oct 29, 2013 at 4:30 PM, Timothy J Massey  wrote:
>
>
> Check lost+found and trash while you're at it and see what's in there.  They 
> should both be empty.
>
> I'm with Jeff:  I think that you have multiple PC trees that are not part of 
> the pool.  How you managed that I'm not sure.  But you need to find those 
> files and clean them up.  Start with Jeff's command and go from there.

This could happen if the backups were originally on a different
filesystem and were copied over without preserving the pool hardlinks.
 For example if you rsync an individual pc directory into place,
subsequent rsync runs will link against those copies for existing
files but will only make the pool links for new/changed files.

-- 
   Les Mikesell
 lesmikes...@gmail.com

--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[backuppc]

Les Mikesell wrote at about 16:51:12 -0500 on Tuesday, October 29, 2013:
 > On Tue, Oct 29, 2013 at 4:30 PM, Timothy J Massey  
 > wrote:
 > >
 > >
 > > Check lost+found and trash while you're at it and see what's in there.  
 > > They should both be empty.
 > >
 > > I'm with Jeff:  I think that you have multiple PC trees that are not part 
 > > of the pool.  How you managed that I'm not sure.  But you need to find 
 > > those files and clean them up.  Start with Jeff's command and go from 
 > > there.
 > 
 > This could happen if the backups were originally on a different
 > filesystem and were copied over without preserving the pool hardlinks.
 >  For example if you rsync an individual pc directory into place,
 > subsequent rsync runs will link against those copies for existing
 > files but will only make the pool links for new/changed files.
 > 
 > -- 

It also can happen if you have filesystems with flaky hard linking --
I once had that issue with a bad user-space nfs module.

--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Craig O'Brien]

The folder /backup is the root of the disk. I mounted the disk there, doing
the ls -l /backup showed all the root folders on the disk. Perhaps there is
something going on with the PC folders, as the lost+found and trash folders
are both empty.

I'm not sure how I can go about determining if a particular backup is using
the pool or just storing the files in the PC folder. What's the best way to
check if a given backup set is represented in the pool or not? Would
knowing the size of all the pc folders help narrow it down?

I'm not sure if this is the best way to check the hard linking, but here's
a test I thought might be helpful. I did this command to see if a common
file in these backups are pointing to the same inodes.

bash-4.1$ ls -i /backup/pc/*/*/ffileshare/fWindows/fexplorer.exe

The output is long so I'll give a snippet:

bash-4.1$ ls -i /backup/pc/*/*/ffileshare/fWindows/fexplorer.exe
635979167 /backup/pc/120p1m1/75/ffileshare/fWindows/fexplorer.exe
 646452561 /backup/pc/7qk56d1/79/ffileshare/fWindows/fexplorer.exe
635979167 /backup/pc/120p1m1/76/ffileshare/fWindows/fexplorer.exe
 646452561 /backup/pc/7qk56d1/80/ffileshare/fWindows/fexplorer.exe
635979167 /backup/pc/327kkn1/87/ffileshare/fWindows/fexplorer.exe
 646452561 /backup/pc/7qk56d1/81/ffileshare/fWindows/fexplorer.exe
635979167 /backup/pc/327kkn1/88/ffileshare/fWindows/fexplorer.exe
 646452561 /backup/pc/7qk56d1/82/ffileshare/fWindows/fexplorer.exe

And it continued like that which shows me that a common file is going to
the same inodes in these backups which tells me the pool should be working
in theory. (I'm assuming the 2 variants account for different versions of
windows.)

So I'm pretty stumped at how to figure out what happened to it.


Regards,
Craig


On Tue, Oct 29, 2013 at 6:07 PM,  wrote:

> Les Mikesell wrote at about 16:51:12 -0500 on Tuesday, October 29, 2013:
>  > On Tue, Oct 29, 2013 at 4:30 PM, Timothy J Massey 
> wrote:
>  > >
>  > >
>  > > Check lost+found and trash while you're at it and see what's in
> there.  They should both be empty.
>  > >
>  > > I'm with Jeff:  I think that you have multiple PC trees that are not
> part of the pool.  How you managed that I'm not sure.  But you need to find
> those files and clean them up.  Start with Jeff's command and go from there.
>  >
>  > This could happen if the backups were originally on a different
>  > filesystem and were copied over without preserving the pool hardlinks.
>  >  For example if you rsync an individual pc directory into place,
>  > subsequent rsync runs will link against those copies for existing
>  > files but will only make the pool links for new/changed files.
>  >
>  > --
>
> It also can happen if you have filesystems with flaky hard linking --
> I once had that issue with a bad user-space nfs module.
>
>
> --
> Android is increasing in popularity, but the open development platform that
> developers love is also attractive to malware creators. Download this white
> paper to learn more about secure code signing practices that can help keep
> Android apps secure.
> http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
> ___
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
>
--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Adam Goryachev]

On 30/10/13 11:21, Craig O'Brien wrote:
The folder /backup is the root of the disk. I mounted the disk there, 
doing the ls -l /backup showed all the root folders on the disk. 
Perhaps there is something going on with the PC folders, as the 
lost+found and trash folders are both empty.


I'm not sure how I can go about determining if a particular backup is 
using the pool or just storing the files in the PC folder. What's the 
best way to check if a given backup set is represented in the pool or 
not? Would knowing the size of all the pc folders help narrow it down?


I'm not sure if this is the best way to check the hard linking, but 
here's a test I thought might be helpful. I did this command to see if 
a common file in these backups are pointing to the same inodes.




I'm fairly sure:
du -sm /backup/pool /backup/cpool /backup/pc/*

It should count all the data under pool and cpool, and there should be 
minimal space used for the pc folders (because it counts the space for 
the first time the inode is seen)


The other way I've checked is with "stat filename" which will show the 
number of links to the file.


Regards,
Adam

On Tue, Oct 29, 2013 at 6:07 PM, > wrote:


Les Mikesell wrote at about 16:51:12 -0500 on Tuesday, October 29,
2013:
 > On Tue, Oct 29, 2013 at 4:30 PM, Timothy J Massey
mailto:tmas...@obscorp.com>> wrote:
 > >
 > >
 > > Check lost+found and trash while you're at it and see what's
in there.  They should both be empty.
 > >
 > > I'm with Jeff:  I think that you have multiple PC trees that
are not part of the pool.  How you managed that I'm not sure.  But
you need to find those files and clean them up.  Start with Jeff's
command and go from there.
 >
 > This could happen if the backups were originally on a different
 > filesystem and were copied over without preserving the pool
hardlinks.
 >  For example if you rsync an individual pc directory into place,
 > subsequent rsync runs will link against those copies for existing
 > files but will only make the pool links for new/changed files.
 >
 > --

It also can happen if you have filesystems with flaky hard linking --
I once had that issue with a bad user-space nfs module.



--
Adam Goryachev Website Managers www.websitemanagers.com.au
--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Sharuzzaman Ahmat Raslan]

Have you removed some PC from the backup list?

If you have, the folder to that PC is still available in /backup/pc/ .  You have to remove the folder manually.

I believe that will cause high disk usage, as it is not linking to the pool.

Note at the bottom of Edit Hosts:

To delete a host, hit the Delete button. For Add, Delete, and configuration
copy, changes don't take effect until you select Save. None of the deleted
host's backups will be removed, so if you accidently delete a host, simply
re-add it. *To completely remove a host's backups, you need to manually
remove the files below /var/lib/backuppc/pc/HOST
*


Thanks.




On Wed, Oct 30, 2013 at 8:21 AM, Craig O'Brien  wrote:

> The folder /backup is the root of the disk. I mounted the disk there,
> doing the ls -l /backup showed all the root folders on the disk. Perhaps
> there is something going on with the PC folders, as the lost+found and
> trash folders are both empty.
>
> I'm not sure how I can go about determining if a particular backup is
> using the pool or just storing the files in the PC folder. What's the best
> way to check if a given backup set is represented in the pool or not? Would
> knowing the size of all the pc folders help narrow it down?
>
> I'm not sure if this is the best way to check the hard linking, but here's
> a test I thought might be helpful. I did this command to see if a common
> file in these backups are pointing to the same inodes.
>
> bash-4.1$ ls -i /backup/pc/*/*/ffileshare/fWindows/fexplorer.exe
>
> The output is long so I'll give a snippet:
>
> bash-4.1$ ls -i /backup/pc/*/*/ffileshare/fWindows/fexplorer.exe
> 635979167 /backup/pc/120p1m1/75/ffileshare/fWindows/fexplorer.exe
>  646452561 /backup/pc/7qk56d1/79/ffileshare/fWindows/fexplorer.exe
> 635979167 /backup/pc/120p1m1/76/ffileshare/fWindows/fexplorer.exe
>  646452561 /backup/pc/7qk56d1/80/ffileshare/fWindows/fexplorer.exe
> 635979167 /backup/pc/327kkn1/87/ffileshare/fWindows/fexplorer.exe
>  646452561 /backup/pc/7qk56d1/81/ffileshare/fWindows/fexplorer.exe
> 635979167 /backup/pc/327kkn1/88/ffileshare/fWindows/fexplorer.exe
>  646452561 /backup/pc/7qk56d1/82/ffileshare/fWindows/fexplorer.exe
>
> And it continued like that which shows me that a common file is going to
> the same inodes in these backups which tells me the pool should be working
> in theory. (I'm assuming the 2 variants account for different versions of
> windows.)
>
> So I'm pretty stumped at how to figure out what happened to it.
>
>
> Regards,
> Craig
>
>
> On Tue, Oct 29, 2013 at 6:07 PM,  wrote:
>
>> Les Mikesell wrote at about 16:51:12 -0500 on Tuesday, October 29, 2013:
>>  > On Tue, Oct 29, 2013 at 4:30 PM, Timothy J Massey 
>> wrote:
>>  > >
>>  > >
>>  > > Check lost+found and trash while you're at it and see what's in
>> there.  They should both be empty.
>>  > >
>>  > > I'm with Jeff:  I think that you have multiple PC trees that are not
>> part of the pool.  How you managed that I'm not sure.  But you need to find
>> those files and clean them up.  Start with Jeff's command and go from there.
>>  >
>>  > This could happen if the backups were originally on a different
>>  > filesystem and were copied over without preserving the pool hardlinks.
>>  >  For example if you rsync an individual pc directory into place,
>>  > subsequent rsync runs will link against those copies for existing
>>  > files but will only make the pool links for new/changed files.
>>  >
>>  > --
>>
>> It also can happen if you have filesystems with flaky hard linking --
>> I once had that issue with a bad user-space nfs module.
>>
>>
>> --
>> Android is increasing in popularity, but the open development platform
>> that
>> developers love is also attractive to malware creators. Download this
>> white
>> paper to learn more about secure code signing practices that can help keep
>> Android apps secure.
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
>> ___
>> BackupPC-users mailing list
>> BackupPC-users@lists.sourceforge.net
>> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
>> Wiki:http://backuppc.wiki.sourceforge.net
>> Project: http://backuppc.sourceforge.net/
>>
>
>
>
> --
> Android is increasing in popularity, but the open development platform that
> developers love is also attractive to malware creators. Download this white
> paper to learn more about secure code signing practices that can help keep
> Android apps secure.
> http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
> ___
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:http://backuppc.wiki.sourceforge.net
> Project: http://backu

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Timothy J Massey]

"Craig O'Brien"  wrote on 10/29/2013 08:21:11 PM:

> I'm not sure how I can go about determining if a particular backup 
> is using the pool or just storing the files in the PC folder. What's
> the best way to check if a given backup set is represented in the 
> pool or not? Would knowing the size of all the pc folders help narrow it 
down?

Nope. 

> I'm not sure if this is the best way to check the hard linking, but 
> here's a test I thought might be helpful. I did this command to see 
> if a common file in these backups are pointing to the same inodes.

You want to look for files in the pc directory that have only one 
hardlink.  These files are not in the pool and need to either be deleted 
or connected to files in the pool.

Jeff Kosowsky gave you a command to list files with only one hardlink.  
Adam Goryachev gave you a good du command to find out how much space is 
being taken by the pc directory *after* counting the files in the pool 
separately.  That command will take a long time to run, but it will give 
you a pretty clear idea of where the space is being consumed.

I would not stake my life on this, but I would bet a pretty substantial 
amount of money:  you did something to break the pooling.  Most likely by 
copying backups around.  This undid the hardlinks and left you with 
individual copies of the files.


Or punt completely:  rebuild the BackupPC server and start over.

You could do almost as well by confirming that your latest backups *are* 
hardlinking properly and then deleting all of the old backups except maybe 
a copy or two.  I would not delete the copies by hand, but rather change 
the configuration to only keep 1 full and 1 incremental.  It might be a 
good idea to make some archives to make sure you have a good copy 
somewhere.  In any case, once BackupPC has deleted all of the old backups, 
go into your pc directories and make sure that there is indeed only the 
backups listed in the GUI in the folder structure.  Then, change the 
incremental and full keep counts back to what they should be and allow it 
to rebuild.


The only other thing that I can think of is that you did something wrong 
with archiving and accidentally archived data somewhere within the 
BackupPC tree.  In my case, I archive to a removable hard drive and 
sometimes the drive is not mounted when the archive runs.  The archives 
are then put on the backup drive (because that's where the removable drive 
is mounted).  That's tricky because you can't see the files when the drive 
*is* mounted (which is the vast majority of the time).  I have to unmount 
the drive and then I can see terabytes of archive data that should have 
been written to a removable drive.

I don't know if that might be part of your problem.  But it's the only 
other thing I can think of.

Tim Massey
 
Out of the Box Solutions, Inc. 
Creative IT Solutions Made Simple!
http://www.OutOfTheBoxSolutions.com
tmas...@obscorp.com 
 
22108 Harper Ave.
St. Clair Shores, MI 48080
Office: (800)750-4OBS (4627)
Cell: (586)945-8796 
--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Adam Goryachev]

On 31/10/13 00:04, Timothy J Massey wrote:
> The only other thing that I can think of is that you did something
> wrong with archiving and accidentally archived data somewhere within
> the BackupPC tree.  In my case, I archive to a removable hard drive
> and sometimes the drive is not mounted when the archive runs.  The
> archives are then put on the backup drive (because that's where the
> removable drive is mounted).  That's tricky because you can't see the
> files when the drive *is* mounted (which is the vast majority of the
> time).  I have to unmount the drive and then I can see terabytes of
> archive data that should have been written to a removable drive.
>
> I don't know if that might be part of your problem.  But it's the only
> other thing I can think of. 

Not really relevant to this thread, but I have in the past added a empty
file to each of the removable drives, then test if the file exists
before creating the archives. If the drive isn't mounted, the file won't
exist. Thus preventing that issue.

I'm sure you've probably considered this previously already :)

Regards,
Adam

-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au

--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Timothy J Massey]

Adam Goryachev  wrote on 10/30/2013 
09:18:59 AM:

> Not really relevant to this thread, but I have in the past added a 
> empty file to each of the removable drives, then test if the file 
> exists before creating the archives. If the drive isn't mounted, the
> file won't exist. Thus preventing that issue.
> 
> I'm sure you've probably considered this previously already :)

Thank you for the suggestion!

My thought was to parse the output of "df /path/to/drive" and confirm that 
it was mounted correctly.  (I already do that in the scripts I use to 
mount the removable drive and re-format it.)  If it happened more than 
once or twice a year, I probably would!  :)

Your way certainly works, too.  Because it's the root of the removable 
drive, I could simply look for lost+found, too!  :)

Tim Massey


 
Out of the Box Solutions, Inc. 
Creative IT Solutions Made Simple!
http://www.OutOfTheBoxSolutions.com
tmas...@obscorp.com 
 
22108 Harper Ave.
St. Clair Shores, MI 48080
Office: (800)750-4OBS (4627)
Cell: (586)945-8796 
--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Tyler J. Wagner]

On 2013-10-29 17:53, Craig O'Brien wrote:
> On the General Server Information page, it says "Pool is 2922.42GB
> comprising 6061942 files and 4369 directories," but our pool file system
> which contains nothing but backuppc and is 11 TB in size is 100% full.

Did you forget to exclude the path to TopDir (usually /var/lib/backuppc)
from the backup of the BackupPC server itself? I've seen that before. Heck,
I've DONE that before.

Regards,
Tyler

-- 
"The intellectual is constantly betrayed by his vanity. Godlike he
blandly assumes that he can express everything in words; whereas the
things one loves, lives, and dies for are not, in the last analysis
completely expressible in words."
   -- Anne Morrow Lindbergh

--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Craig O'Brien]

> I'm fairly sure:
> du -sm /backup/pool /backup/cpool /backup/pc/*
> It should count all the data under pool and cpool, and there should be
minimal space used for the pc folders (because it counts the space for the
first time the inode is seen)

I'm trying that now. I'll report back when it finishes.

> To delete a host, hit the Delete button. For Add, Delete, and
configuration copy, changes don't take effect until you select Save. None
of the deleted host's backups will be removed, so if you accidently delete
a host, simply re-add it.
*> To completely remove a host's backups, you need to manually remove the
files below /var/lib/backuppc/pc/HOST *

This is how I've done it when I've removed a host. I would delete the
/backup/pc/host directory and remove the entry from /etc/BackupPC/hosts
file.

> I would not stake my life on this, but I would bet a pretty substantial
amount of money:  you did something to break the pooling.  Most likely by
copying backups around.  This undid the hardlinks and > left you with
individual copies of the files.

I don't doubt the pooling is probably broken but I haven't moved any
backups around. For what it's worth I before I switched all the pc's to use
rsync instead of smb a couple months ago my pool file system was sitting at
30%. I don't know if that's relevant but it does seem odd that my problems
seem to have started with that.

> Or punt completely:  rebuild the BackupPC server and start over.

> You could do almost as well by confirming that your latest backups *are*
hardlinking properly and then deleting all of the old backups except maybe
a copy or two.
> I would not delete the copies by > hand, but rather change the
configuration to only keep 1 full and 1 incremental.
> It might be a good idea to make some archives to make sure you have a
good copy somewhere.  In any case, once BackupPC has deleted all of the old
backups,
> go into your pc directories and make sure that there is indeed only the
backups listed in the GUI in the folder structure.
> Then, change the incremental and full keep counts back to what they
should be and allow it to rebuild.

I'll probably have to do that. At this point I'm just trying to add to the
knowledge base and figure out how it went wrong so it doesn't just happen
again.

> My thought was to parse the output of "df /path/to/drive" and confirm
that it was mounted correctly.

Just in case it helps at all:

bash-4.1$ df -h /backup
FilesystemSize  Used Avail Use% Mounted on
/dev/sda1  11T  9.7T   28G 100% /backup
bash-4.1$

> Did you forget to exclude the path to TopDir (usually /var/lib/backuppc)
> from the backup of the BackupPC server itself? I've seen that before.
Heck,
> I've DONE that before.

I don't have the server backing itself up.


Here's my config file (with #comment lines removed) just in case that helps
at all.
-

$Conf{ServerHost} = 'localhost';
$Conf{ServerPort} = -1;
$Conf{ServerMesgSecret} = '';
$Conf{MyPath} = '/bin';
$Conf{UmaskMode} = 23;
$Conf{WakeupSchedule} = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15,
16, 17, 18, 19, 20, 21, 22, 23];
$Conf{MaxBackups} = 4;
$Conf{MaxUserBackups} = 4;
$Conf{MaxPendingCmds} = 15;
$Conf{CmdQueueNice} = 10;
$Conf{MaxBackupPCNightlyJobs} = 4;
$Conf{BackupPCNightlyPeriod} = 1;
$Conf{MaxOldLogFiles} = 14;
$Conf{DfPath} = '/bin/df';
$Conf{DfCmd} = '$dfPath $topDir';
$Conf{SplitPath} = '/usr/bin/split';
$Conf{ParPath} = undef;
$Conf{CatPath} = '/bin/cat';
$Conf{GzipPath} = '/bin/gzip';
$Conf{Bzip2Path} = '/usr/bin/bzip2';
$Conf{DfMaxUsagePct} = 95;
$Conf{TrashCleanSleepSec} = 300;
$Conf{DHCPAddressRanges} = [];
$Conf{BackupPCUser} = 'backuppc';
$Conf{TopDir} = '/var/lib/BackupPC/';
$Conf{ConfDir} = '/etc/BackupPC/';
$Conf{LogDir} = '/var/log/BackupPC';
$Conf{InstallDir} = '/usr/share/BackupPC';
$Conf{CgiDir} = '/usr/share/BackupPC/sbin/';
$Conf{BackupPCUserVerify} = '1';
$Conf{HardLinkMax} = 31999;
$Conf{PerlModuleLoad} = undef;
$Conf{ServerInitdPath} = undef;
$Conf{ServerInitdStartCmd} = '';
$Conf{FullPeriod} = 90;
$Conf{IncrPeriod} = 7;
$Conf{FullKeepCnt} = [ 4 ];
$Conf{FullKeepCntMin} = 1;
$Conf{FullAgeMax} = 3000;
$Conf{IncrKeepCnt} = 13;
$Conf{IncrKeepCntMin} = 1;
$Conf{IncrAgeMax} = 45;
$Conf{IncrLevels} = [ 1 ];
$Conf{BackupsDisable} = 0;
$Conf{PartialAgeMax} = 3;
$Conf{IncrFill} = '0';
$Conf{RestoreInfoKeepCnt} = 5;
$Conf{ArchiveInfoKeepCnt} = 5;
$Conf{BackupFilesOnly} = {};
$Conf{BackupFilesExclude} = {};
$Conf{BlackoutBadPingLimit} = 3;
$Conf{BlackoutGoodCnt} = 7;
$Conf{BlackoutPeriods} = [
  {
'hourEnd' => '19.5',
'weekDays' => [
  1,
  2,
  3,
  4,
  5
],
'hourBegin' => 7
  }
];
$Conf{BackupZeroFilesIsFatal} = '1';
$Conf{XferMethod} = 'rsyncd';
$Conf{XferLogLevel} = 1;
$Conf{ClientCharset} = '';
$Conf{ClientCharsetLegacy} = 'iso-8859-1';
$Conf{SmbShareName} = [
  'C$'
];
$Conf{SmbShareUserName} = '';
$Conf{SmbSharePasswd} = '';
$Conf{SmbClientPath} = '/usr/bin/smbclient';

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Holger Parplies]

Hi,

I'll reply here, because I think the issue is visible here.

Craig O'Brien wrote on 2013-10-29 15:30:46 -0400 [Re: [BackupPC-users] Disk 
space used far higher than reported pool size]:
> The topdir is /var/lib/BackupPC which is a link to /backup

I believe that may be your problem. I'm not sure why, but I vaguely recall
reports of linking problems related to softlinking the pool directory (though
I'm not absolutely sure about the exact circumstances, hence "vaguely
recall" ;-), though *in theory* it should work. I'd always prefer mounting the
partition to /var/lib/backuppc instead of the softlink if you don't have a
*very* good reason for requiring the pool to be mounted elsewhere.

Like Tim, I also wouldn't bet my life on it, but I'm fairly sure you'll find
large amounts of "BackupPC_link got error -4 when calling MakeFileLink"
messages in your log files.

Also note that at least for *rsync backups* files will be hardlinked to
identical copies in previous backups even if pooling isn't working.

Hope that helps, and wish I'd found the time yesterday,

Regards,
Holger

--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Les Mikesell]

On Wed, Oct 30, 2013 at 10:12 AM, Holger Parplies  wrote:
>
> Craig O'Brien wrote on 2013-10-29 15:30:46 -0400 [Re: [BackupPC-users] Disk 
> space used far higher than reported pool size]:
>> The topdir is /var/lib/BackupPC which is a link to /backup
>
> I believe that may be your problem. I'm not sure why, but I vaguely recall
> reports of linking problems related to softlinking the pool directory (though
> I'm not absolutely sure about the exact circumstances, hence "vaguely
> recall" ;-), though *in theory* it should work. I'd always prefer mounting the
> partition to /var/lib/backuppc instead of the softlink if you don't have a
> *very* good reason for requiring the pool to be mounted elsewhere.
>
> Like Tim, I also wouldn't bet my life on it, but I'm fairly sure you'll find
> large amounts of "BackupPC_link got error -4 when calling MakeFileLink"
> messages in your log files.

If you can find those errors in the logs for files that still exist
you might try doing the same link with the ln command to see if it
gives a better diagnostic for why it fails.   It could be something
quirky like a small limit to the number of hardlinks allowed in your
target filesystem, or something you are missing about
permissions/ownership  (nfsv4 can be weird).

-- 
   Les Mikesell
 lesmikes...@gmail.com

--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Holger Parplies]

Just to add two things ...

Holger Parplies wrote on 2013-10-30 16:12:02 +0100 [Re: [BackupPC-users] Disk 
space used far higher than reported pool size]:
> [...]
> Like Tim, I also wouldn't bet my life on it, but I'm fairly sure you'll find
> large amounts of "BackupPC_link got error -4 when calling MakeFileLink"
> messages in your log files.

That would be the main server log file ($topDir/log/LOG and the rotated older
copies), I guess.

> Also note that at least for *rsync backups* files will be hardlinked to
> identical copies in previous backups even if pooling isn't working.

Since you have just written that you are, in fact, using rsync, I should add
that this will make recovery more difficult, since unchanged files will
continue to be just linked to the version in the reference backup. This file
should normally already be linked to the pool, and BackupPC makes no further
effort to check and fix that.

This means that if you delete all but a few recent backups after fixing the
root cause of the problem, only newly changed files will be added to the pool,
while unchanged files will remain outside the pool. This may or may not be a
problem for you. If it is, you will need to fix pooling for (some) existing
backups, which is Hard(tm) (i.e. costly in terms of CPU power).


Jeffrey, I think we need a script to check pooling? My (still unfinished)
BackupPC_copyPool can generate a (huge) list of files, which can be sort(1)ed
by inode number. Parsing that should easily reveal anything not correctly
linked in an acceptable time frame (of course *generating* the list takes one
traversal of all pool and pc directories, but the rest would be fast enough).
Does that help, or have you already got something more suited? Are you
interested or should I be? ;-)

Regards,
Holger

--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Les Mikesell]

On Wed, Oct 30, 2013 at 10:48 AM, Holger Parplies  wrote:
>
>> Also note that at least for *rsync backups* files will be hardlinked to
>> identical copies in previous backups even if pooling isn't working.
>
> Since you have just written that you are, in fact, using rsync, I should add
> that this will make recovery more difficult, since unchanged files will
> continue to be just linked to the version in the reference backup. This file
> should normally already be linked to the pool, and BackupPC makes no further
> effort to check and fix that.
>
> This means that if you delete all but a few recent backups after fixing the
> root cause of the problem, only newly changed files will be added to the pool,
> while unchanged files will remain outside the pool. This may or may not be a
> problem for you. If it is, you will need to fix pooling for (some) existing
> backups, which is Hard(tm) (i.e. costly in terms of CPU power).

If you can free up some space to work, you could try renaming the
pc/host directories one at a time to hold a copy that you could access
in an emergency and let it start over with a fresh full run where all
files would be new and linked to the pool.  Once you have sufficient
history in the new tree, you can delete the old one that you renamed.
This should eventually clean things up as long as you don't continue
to have link errors.   Alternatively, if you don't need more than the
last backup for one or more targets, you could archive it with the
archive host setup or running Backuppc_tarCreate on to some other
media, delete the host and add it back to get a clean start.

-- 
   Les Mikesell
 lesmikes...@gmail.com

--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Holger Parplies]

Hi,

Les Mikesell wrote on 2013-10-30 11:28:26 -0500 [Re: [BackupPC-users] Disk 
space used far higher than reported pool size]:
> On Wed, Oct 30, 2013 at 10:48 AM, Holger Parplies  wrote:
> >> Also note that at least for *rsync backups* files will be hardlinked to
> >> identical copies in previous backups even if pooling isn't working.
> > [...]
> > This means that if you delete all but a few recent backups after fixing the
> > root cause of the problem, only newly changed files will be added to the
> > pool, while unchanged files will remain outside the pool. This may or may
> > not be a problem for you. If it is, you will need to fix pooling for (some)
> > existing backups, which is Hard(tm) (i.e. costly in terms of CPU power).

I intentionally did not go into detail, because we have not yet confirmed
that this is indeed the problem, and if so, what the requirements of the OP
are.

> If you can free up some space to work, you could try renaming the
> pc/host directories one at a time to hold a copy that you could access
> in an emergency and let it start over with a fresh full run where all
> files would be new and linked to the pool.

Correct, but with an already almost full pool file system you *will* run into
problems, because even unchanged files will now require a new copy. "Some
space" might be a bit of an understatement :).

> [...]
> This should eventually clean things up as long as you don't continue
> to have link errors.

Yes, as it's basically an extension of "start off fresh" with the addition of
"keep old history around in parallel". The notable thing is that you need to
*make sure* you have eliminated the problem for there to be any point in
starting over.

Aside from that, I would think it might be worth the effort of determining
whether all hosts are affected or not (though I can't really see why there
should be a difference between hosts). If some aren't, you could at least keep
their history.

Regards,
Holger

--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Adam Goryachev]

On 31/10/13 07:51, Holger Parplies wrote:
> Yes, as it's basically an extension of "start off fresh" with the addition of
> "keep old history around in parallel". The notable thing is that you need to
> *make sure* you have eliminated the problem for there to be any point in
> starting over.
>
> Aside from that, I would think it might be worth the effort of determining
> whether all hosts are affected or not (though I can't really see why there
> should be a difference between hosts). If some aren't, you could at least keep
> their history.
I suspect at least some hosts OR some backups are correct, or else OP 
wouldn't have anything in the pool. If you find the problem affects all 
hosts (the du command discussed previously will tell you that), then you 
might want to look at one individual host like this:
du -sm /backup/pool /backup/cpool /backup/pc/host1/*

This should be a *lot* quicker than the previous du command, and also 
should show minimal disk usage for each backup for host1. It is quicker 
because you are only looking at the set of files for the pool, plus one 
host.

PS, at this stage, you may want to look at the recent thread regarding 
disk caches, and caching directory entries instead of file contents. It 
might help with all the directory based searches you are doing to find 
the problem. Long term you may (or not) want to keep the settings.

Regards,
Adam

-- 
Adam Goryachev Website Managers www.websitemanagers.com.au

--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[backuppc]

Holger Parplies wrote at about 16:48:11 +0100 on Wednesday, October 30, 2013:
 > Jeffrey, I think we need a script to check pooling? My (still unfinished)
 > BackupPC_copyPool can generate a (huge) list of files, which can be sort(1)ed
 > by inode number. Parsing that should easily reveal anything not correctly
 > linked in an acceptable time frame (of course *generating* the list takes one
 > traversal of all pool and pc directories, but the rest would be fast enough).
 > Does that help, or have you already got something more suited? Are you
 > interested or should I be? ;-)
 

I have code that will do this in 2 related ways:

1. Run my routine "BackupPC_copyPcPool.pl" with the "--fixlinks|-f"
   option which will fix missing (or invalid) pc-to-pool links on the
   fly as the routine crawls the pc tree (after creating an in-memory
   hash cache of the pool inodes)

3. Run my routine "BackupPC_copyPcPool.pl" to generate a list of the
   non-zero length, non-linked files (other than backupInfo which is
   the only non-zero length file in the pc tree that should not be
   linked to the pool) which lie in the pc tree. The routine always
   creates this list since these files would need to be transferred
   manually if not linked to the pool.

   Then pipe this list of unlinked files to my routine
   "BackupPc_relinkPcFiles.pl" to fix each of the non-linked files

Note that BackupPC_copyPcPool works by caching the inode numbers of
the pool/cpool entries in a hash which allows for quick lookup and
checking of whether a pc tree file is linked to a valid pool file.

Note that the above methods take care of the cases when pc tree
files are:
 1. Unlinked to anything else (nlinks =1)
 2. Linked to other pc files (but not to the pool)

Note that both methods properly either make a new link in the pool or
delete the existing pc file and link it to a pre-existing pool entry
depending on whether or not a pool entry already exists.


--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Holger Parplies]

Hi,

Adam Goryachev wrote on 2013-10-31 09:04:48 +1100 [Re: [BackupPC-users] Disk 
space used far higher than reported pool size]:
> On 31/10/13 07:51, Holger Parplies wrote:
> > [...]
> > Aside from that, I would think it might be worth the effort of determining
> > whether all hosts are affected or not (though I can't really see why there
> > should be a difference between hosts). If some aren't, you could at least
> > keep their history.
> I suspect at least some hosts OR some backups are correct, or else OP 
> wouldn't have anything in the pool.

as I understand it, the backups from before the change from smb to rsyncd are
linked into the pool. Since the change, some or all are not. Whether the
change of XferMethod has anything to do with the problem or whether it
coincidentally happened at about the same point in time remains to be seen.
I still suspect the link to $topDir as cause, and BackupPC_link is independent
of the XferMethod used (so a change in XferMethod shouldn't have any influence).

> [...] you might want to look at one individual host like this:
> du -sm /backup/pool /backup/cpool /backup/pc/host1/*
> 
> This should be a *lot* quicker than the previous du command, and also 
> should show minimal disk usage for each backup for host1. It is quicker 
> because you are only looking at the set of files for the pool, plus one 
> host.

Just keep in mind that *incrementals* might be small even if not linked to
pool files.

Oh, and there is still another method that is *orders of magnitude* faster:
look into the log file(s), or even at the *size* of the log files. If it
happens every day, for each host, it shouldn't be hard to find. You can even
write a Perl one-liner to show you which hosts it happens for (give me a
sample log line and I will).

If the log files show nothing, we're back to finding the problem, but I doubt
that. You can't "break pooling" by copying, as was suggested. Yes, you get
independent copies of files, and they might stay independent, but changed
files should get pooled again, and your file system usage wouldn't continue
growing in such a way as it seems to be. If pooling is currently "broken",
there's a reason for that, and there should be log messages indicating
problems.

> PS, at this stage, you may want to look at the recent thread regarding 
> disk caches, and caching directory entries instead of file contents. It 
> might help with all the directory based searches you are doing to find 
> the problem. Long term you may (or not) want to keep the settings.

Yes, but remember that for a similarly sized pool it used up about 32 GB of
96 GB available memory. If you can do your investigation on a reasonably idle
system (i.e. not running backups, without long pauses), you should get all the
benefits of caching your amount of memory allows without any tuning. And even
tuning won't let you hold 32 GB of file system metadata in 4 GB of memory :-).
It all depends on file count and hardware memory configuration.

Regards,
Holger

--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Craig O'Brien]

The du -hs /backup/pool /backup/cpool /backup/pc/* has finished. Basically
I had 1 host that was taking up 6.9 TB of data with 2.8 TB in the cpool
directory and most of the other hosts averaging a GB each.

The 1 host was our file server (which I happen to know has a 2 TB volume
(1.3 TB currently used) that is our main fileshare.

I looked through the error log for this pc on backups with the most errors
and found thousands of these:

Unable to read 8388608 bytes from
/var/lib/BackupPC//pc/myfileserver/new//ffileshare/RStmp got=0,
seekPosn=1501757440 (0,512,147872,1499463680,2422719488)
Unable to read 8388608 bytes from
/var/lib/BackupPC//pc/myfileserver/new//ffileshare/RStmp got=0,
seekPosn=1501757440 (0,512,147872,1499463680,2422719488)
Unable to read 8388608 bytes from
/var/lib/BackupPC//pc/myfileserver/new//ffileshare/RStmp got=0,
seekPosn=1501757440 (0,512,147872,1499463680,2422719488)
Unable to read 8388608 bytes from
/var/lib/BackupPC//pc/myfileserver/new//ffileshare/RStmp got=0,
seekPosn=1501757440 (0,512,147872,1499463680,2422719488)
Unable to read 8388608 bytes from
/var/lib/BackupPC//pc/myfileserver/new//ffileshare/RStmp got=0,
seekPosn=1501757440 (0,512,147872,1499463680,2422719488)
Unable to read 8388608 bytes from
/var/lib/BackupPC//pc/myfileserver/new//ffileshare/RStmp got=0,
seekPosn=1501757440 (0,512,147872,1499463680,2422719488)
Unable to read 8388608 bytes from
/var/lib/BackupPC//pc/myfileserver/new//ffileshare/RStmp got=0,
seekPosn=1501757440 (0,512,147872,1499463680,2422719488)
Unable to read 8388608 bytes from
/var/lib/BackupPC//pc/myfileserver/new//ffileshare/RStmp got=0,
seekPosn=1501757440 (0,512,147872,1499463680,2422719488)
Unable to read 8388608 bytes from
/var/lib/BackupPC//pc/myfileserver/new//ffileshare/RStmp got=0,
seekPosn=1501757440 (0,512,147872,1499463680,2422719488)
Unable to read 8388608 bytes from
/var/lib/BackupPC//pc/myfileserver/new//ffileshare/RStmp got=0,
seekPosn=1501757440 (0,512,147872,1499463680,2422719488)
Unable to read 8388608 bytes from
/var/lib/BackupPC//pc/myfileserver/new//ffileshare/RStmp got=0,
seekPosn=1501757440 (0,512,147872,1499463680,2422719488)

I didn't see any of the ""BackupPC_link got error -4" errors. So now I'm
running this command:

du -hs /backup/pool /backup/cpool /backup/pc/myfileserver/*

to see which backups are doing the most damage. I'll report back once that
finishes.

Thanks for all your help!


Regards,
Craig


On Wed, Oct 30, 2013 at 10:24 PM, Holger Parplies  wrote:

> Hi,
>
> Adam Goryachev wrote on 2013-10-31 09:04:48 +1100 [Re: [BackupPC-users]
> Disk space used far higher than reported pool size]:
> > On 31/10/13 07:51, Holger Parplies wrote:
> > > [...]
> > > Aside from that, I would think it might be worth the effort of
> determining
> > > whether all hosts are affected or not (though I can't really see why
> there
> > > should be a difference between hosts). If some aren't, you could at
> least
> > > keep their history.
> > I suspect at least some hosts OR some backups are correct, or else OP
> > wouldn't have anything in the pool.
>
> as I understand it, the backups from before the change from smb to rsyncd
> are
> linked into the pool. Since the change, some or all are not. Whether the
> change of XferMethod has anything to do with the problem or whether it
> coincidentally happened at about the same point in time remains to be seen.
> I still suspect the link to $topDir as cause, and BackupPC_link is
> independent
> of the XferMethod used (so a change in XferMethod shouldn't have any
> influence).
>
> > [...] you might want to look at one individual host like this:
> > du -sm /backup/pool /backup/cpool /backup/pc/host1/*
> >
> > This should be a *lot* quicker than the previous du command, and also
> > should show minimal disk usage for each backup for host1. It is quicker
> > because you are only looking at the set of files for the pool, plus one
> > host.
>
> Just keep in mind that *incrementals* might be small even if not linked to
> pool files.
>
> Oh, and there is still another method that is *orders of magnitude* faster:
> look into the log file(s), or even at the *size* of the log files. If it
> happens every day, for each host, it shouldn't be hard to find. You can
> even
> write a Perl one-liner to show you which hosts it happens for (give me a
> sample log line and I will).
>
> If the log files show nothing, we're back to finding the problem, but I
> doubt
> that. You can't "break pooling" by copying, as was suggested. Yes, you get
> independent copies of files, and they might stay independent, but changed
> files should get pooled again, and your file system usage wouldn't continue
> growing in such a way as it seems to 

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Les Mikesell]

On Thu, Oct 31, 2013 at 7:49 AM, Craig O'Brien  wrote:
>
> Unable to read 8388608 bytes from
> /var/lib/BackupPC//pc/myfileserver/new//ffileshare/RStmp got=0,

What is the underlying storage here - nfs?

-- 
   Les Mikesell
 lesmikes...@gmail.com

--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Craig O'Brien]

> What is the underlying storage here - nfs?

Local SATA disks in a RAID 5 (5 disks, 3TB each in capacity)

Regards,
Craig


On Thu, Oct 31, 2013 at 10:37 AM, Les Mikesell wrote:

> On Thu, Oct 31, 2013 at 7:49 AM, Craig O'Brien 
> wrote:
> >
> > Unable to read 8388608 bytes from
> > /var/lib/BackupPC//pc/myfileserver/new//ffileshare/RStmp got=0,
>
> What is the underlying storage here - nfs?
>
> --
>Les Mikesell
>  lesmikes...@gmail.com
>
>
> --
> Android is increasing in popularity, but the open development platform that
> developers love is also attractive to malware creators. Download this white
> paper to learn more about secure code signing practices that can help keep
> Android apps secure.
> http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
> ___
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
>
--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Les Mikesell]

On Thu, Oct 31, 2013 at 9:47 AM, Craig O'Brien  wrote:
>> What is the underlying storage here - nfs?
>
> Local SATA disks in a RAID 5 (5 disks, 3TB each in capacity)

I think I'd force an fsck just on general principles even though it
will take a long time to complete.   Google turns up a few hits on
similar problems, but I don't see a definitive answer.   RStmp is
supposed to be used to hold an uncompressed copy of the previous
version of a large file with changes so rsync can seek to match up the
changed block positions, so this error probably has something to do
with your compressed copy being corrupted and not uncompressing
properly.

-- 
  Les Mikesell
 lesmikes...@gmail.com

--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Timothy J Massey]

"Craig O'Brien"  wrote on 10/31/2013 08:49:15 AM:

> The du -hs /backup/pool /backup/cpool /backup/pc/* has finished. 
> Basically I had 1 host that was taking up 6.9 TB of data with 2.8 TB
> in the cpool directory and most of the other hosts averaging a GB each.

Well, there's your problem.

> The 1 host was our file server (which I happen to know has a 2 TB 
> volume (1.3 TB currently used) that is our main fileshare. 
> 
> I looked through the error log for this pc on backups with the most 
> errors and found thousands of these: 

Just out of curiosity, why hadn't you already done that?!?

> Unable to read 8388608 bytes from /var/lib/BackupPC//pc/
> myfileserver/new//ffileshare/RStmp got=0, seekPosn=1501757440 (0,
> 512,147872,1499463680,2422719488)

Interesting.  I'd make sure that the filesystem is OK before I went much 
farther...  Stop BackupPC, unmount /backup and fsck /dev/

> du -hs /backup/pool /backup/cpool /backup/pc/myfileserver/* 
> 
> to see which backups are doing the most damage. I'll report back 
> once that finishes.

With that, you should be able to find the bakup number(s) that are not 
linked.  You can delete them and free up space.

The big question is, though, why they aren't linking.  I'd really start at 
the bottom of the stack (the physical drives) and work your way up.  Check 
dmesg for any hardware errors.  fsck the filesystem.  Did I read correctly 
that this is connected vis NFSv4?  I sure hope not...  (I'm willing to 
admit it's a phobia, but there's no *WAY* I would trust my backup to work 
across NFS...)

Tim Massey
 
Out of the Box Solutions, Inc. 
Creative IT Solutions Made Simple!
http://www.OutOfTheBoxSolutions.com
tmas...@obscorp.com 
 
22108 Harper Ave.
St. Clair Shores, MI 48080
Office: (800)750-4OBS (4627)
Cell: (586)945-8796 
--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Timothy J Massey]

Holger Parplies  wrote on 10/30/2013 10:24:05 PM:

> as I understand it, the backups from before the change from smb to 
rsyncd are
> linked into the pool. Since the change, some or all are not. Whether the
> change of XferMethod has anything to do with the problem or whether it
> coincidentally happened at about the same point in time remains to be 
seen.
> I still suspect the link to $topDir as cause, and BackupPC_link is 
independent
> of the XferMethod used (so a change in XferMethod shouldn't have any
> influence).

To add my anecdote, I use a symbolic link for all of my BackupPC hosts:  a 
couple dozen?  And they all work fine.  It's been my standard procedure 
for almost as long as I've been using BackupPC.

Example: 
ls -l /var/lib
lrwxrwxrwx. 1 rootroot 22 Apr 22  2013 BackupPC -> 
/data/BackupPC/TopDir/

mount
/dev/sda1 on /data type ext4 (rw)

I understand phobias from earlier problems (see my earlier e-mail about my 
thoughts on NFS and backups...) but I do not think this one is an issue.


> If the log files show nothing, we're back to finding the problem, but I 
doubt
> that. You can't "break pooling" by copying, as was suggested. Yes, you 
get
> independent copies of files, and they might stay independent, but 
changed
> files should get pooled again, and your file system usage wouldn't 
continue
> growing in such a way as it seems to be. If pooling is currently 
"broken",
> there's a reason for that, and there should be log messages indicating
> problems.

You are 100% correct;  but it depends on how you define "break".  Making a 
copy of a backup will absolutely break pooling--for the copy you just 
made!  :)

It won't prevent *future* copies from pooling, certainly.  But it sure can 
fill up a drive:  even if pooling *is* working correctly for new copies, 
they can still fill up the drive *and* BackupPC_nightly won't do a thing 
about it.

Tim Massey


 
Out of the Box Solutions, Inc. 
Creative IT Solutions Made Simple!
http://www.OutOfTheBoxSolutions.com
tmas...@obscorp.com 
 
22108 Harper Ave.
St. Clair Shores, MI 48080
Office: (800)750-4OBS (4627)
Cell: (586)945-8796 
--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Marcel Meckel]

Hi,

> Example: 
> ls -l /var/lib
> lrwxrwxrwx. 1 rootroot 22 Apr 22  2013 BackupPC -> 
> /data/BackupPC/TopDir/
> 
> mount
> /dev/sda1 on /data type ext4 (rw)

out of curiosity - why don't you just configure /data/BackupPC/TopDir
in config.pl as the TopDir?

Regards
Marcel

-- 
Registrierter Linux User #307343

--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Les Mikesell]

On Thu, Oct 31, 2013 at 11:36 AM, Marcel Meckel
 wrote:
> Hi,
>
>> Example:
>> ls -l /var/lib
>> lrwxrwxrwx. 1 rootroot 22 Apr 22  2013 BackupPC ->
>> /data/BackupPC/TopDir/
>>
>> mount
>> /dev/sda1 on /data type ext4 (rw)
>
> out of curiosity - why don't you just configure /data/BackupPC/TopDir
> in config.pl as the TopDir?

Versions earlier than 3.2 didn't allow that after the initial install
- and in distribution-packaged version (rpm/deb) the location decision
had already been made by the packagers.

-- 
   Les Mikesell
 lesmikes...@gmail.com

--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Craig O'Brien]

> Just out of curiosity, why hadn't you already done that?!?

I didn't know which host was the problem and didn't think of it. Although
I'll readily admit it seems painfully obvious to me now. :)

>The big question is, though, why they aren't linking.  I'd really start at
the bottom of the stack (the physical drives) and work your way up.  Check
dmesg for any hardware errors.


bash-4.1$ grep -i backup /var/log/dmesg*
bash-4.1$

bash-4.1$ grep -i backup /var/log/messages*
messages-20131006:Sep 30 13:53:24 servername kernel: BackupPC_dump[15365]:
segfault at a80 ip 00310f695002 sp 7fff438c9770 error 4 in
libperl.so[310f60+162000]
messages-20131006:Sep 30 13:53:27 servername abrtd: Package 'BackupPC'
isn't signed with proper key
messages-20131020:Oct 19 01:24:54 servername kernel: INFO: task
BackupPC_dump:11922 blocked for more than 120 seconds.
messages-20131020:Oct 19 01:24:54 servername kernel: BackupPC_dump D
0001 0 11922  10626 0x0080
messages-20131020:Oct 19 01:30:54 servername kernel: INFO: task
BackupPC_dump:11922 blocked for more than 120 seconds.
messages-20131020:Oct 19 01:30:54 servername kernel: BackupPC_dump D
0001 0 11922  10626 0x0080
messages-20131020:Oct 19 01:32:54 servername kernel: INFO: task
BackupPC_dump:11922 blocked for more than 120 seconds.
messages-20131020:Oct 19 01:32:54 servername kernel: BackupPC_dump D
0001 0 11922  10626 0x0080
messages-20131020:Oct 19 01:32:54 servername kernel: INFO: task
BackupPC_nightl:18390 blocked for more than 120 seconds.
messages-20131020:Oct 19 01:32:54 servername kernel: BackupPC_nigh D
0001 0 18390   1262 0x0080
messages-20131020:Oct 19 01:48:54 servername kernel: INFO: task
BackupPC_dump:11922 blocked for more than 120 seconds.
messages-20131020:Oct 19 01:48:54 servername kernel: BackupPC_dump D
0003 0 11922  10626 0x0080
messages-20131020:Oct 19 01:52:54 servername kernel: INFO: task
BackupPC_dump:11922 blocked for more than 120 seconds.
messages-20131020:Oct 19 01:52:54 servername kernel: BackupPC_dump D
0001 0 11922  10626 0x0080
messages-20131020:Oct 19 01:52:54 servername kernel: INFO: task
BackupPC_nightl:18390 blocked for more than 120 seconds.
messages-20131020:Oct 19 01:52:54 servername kernel: BackupPC_nigh D
0001 0 18390   1262 0x0080
messages-20131020:Oct 19 01:56:54 servername kernel: INFO: task
BackupPC_dump:11922 blocked for more than 120 seconds.
messages-20131020:Oct 19 01:56:54 servername kernel: BackupPC_dump D
0003 0 11922  10626 0x0080
messages-20131020:Oct 19 02:10:54 servername kernel: INFO: task
BackupPC_dump:11922 blocked for more than 120 seconds.
messages-20131020:Oct 19 02:10:54 servername kernel: BackupPC_dump D
0001 0 11922  10626 0x0080
messages-20131020:Oct 19 02:12:54 servername kernel: INFO: task
BackupPC_dump:11922 blocked for more than 120 seconds.
messages-20131020:Oct 19 02:12:54 servername kernel: BackupPC_dump D
0001 0 11922  10626 0x0080
messages-20131027:Oct 23 09:00:02 servername abrtd: Package 'BackupPC'
isn't signed with proper key


> fsck the filesystem.

bash-4.1$ fsck /dev/sda1
fsck from util-linux-ng 2.17.2
e2fsck 1.41.12 (17-May-2010)
/dev/sda1: clean, 20074506/2929688576 files, 2775975889/2929686016 blocks
bash-4.1$

>Did I read correctly that this is connected vis NFSv4?  I sure hope not...
 (I'm willing to admit it's a phobia, but there's no *WAY* I would trust my
backup to work across NFS...)

The drives are local SATA ones that I set up in a raid 5, directly mounted.
Def not NFS. I had an unrelated drive mounted via NFS, but that had nothing
to do with my backup system and that's probably the source of confusion.

So the du command finished, here's the result:

bash-4.1$ du -hs /backup/pool /backup/cpool /backup/pc/fileserver/*
4.0K/backup/pool
2.8T/backup/cpool
350M/backup/pc/fileserver/223
361M/backup/pc/fileserver/250
373M/backup/pc/fileserver/278
325M/backup/pc/fileserver/302
329M/backup/pc/fileserver/331
330M/backup/pc/fileserver/360
335M/backup/pc/fileserver/388
338M/backup/pc/fileserver/417
345M/backup/pc/fileserver/446
346M/backup/pc/fileserver/475
350M/backup/pc/fileserver/503
450M/backup/pc/fileserver/524
437M/backup/pc/fileserver/525
437M/backup/pc/fileserver/526
437M/backup/pc/fileserver/527
2.5G/backup/pc/fileserver/528
1.4T/backup/pc/fileserver/529
438M/backup/pc/fileserver/530
467M/backup/pc/fileserver/531
438M/backup/pc/fileserver/532
438M/backup/pc/fileserver/533
1.4T/backup/pc/fileserver/534
438M/backup/pc/fileserver/535
1013M   /backup/pc/fileserver/536
442M/backup/pc/fileserver/537
441M/backup/pc/fileserver/538
441M/backup/pc/fileserver/539
1.4T/backup/pc/fileserver/540
441M/backup/pc/fileserver/541
442M/backup/pc/fileserver/542
442M/bac

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Sharuzzaman Ahmat Raslan]

On Fri, Nov 1, 2013 at 1:33 AM, Craig O'Brien  wrote:

> messages-20131006:Sep 30 13:53:24 servername kernel: BackupPC_dump[15365]:
> segfault at a80 ip 00310f695002 sp 7fff438c9770 error 4 in
> libperl.so[310f60+162000]


This error shows BackupPC_dump segfault, and pointing to libperl.so

How do you install your BackupPC ? From source or from RPM?

If from RPM, which repo that you use?

Thanks

-- 
Sharuzzaman Ahmat Raslan
--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Les Mikesell]

On Thu, Oct 31, 2013 at 12:33 PM, Craig O'Brien  wrote:
>
>> fsck the filesystem.
>
> bash-4.1$ fsck /dev/sda1
> fsck from util-linux-ng 2.17.2
> e2fsck 1.41.12 (17-May-2010)
> /dev/sda1: clean, 20074506/2929688576 files, 2775975889/2929686016 blocks
> bash-4.1$

That tells you it was unmounted cleanly last time, not that everything
checks out OK.   Try it with the -f option to make it do the actual
checks.

>
> I don't suppose this helps give any insight to what happened? Thanks for all
> your help!

I think it is related to that RStmp file that isn't uncompressing
correctly so rsync can merge the changes - I'm not sure what happens
after that error, though, or how to find the compressed file that is
probably causing it.

-- 
   Les Mikesell
 lesmikes...@gmail.com

--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Timothy J Massey]

"Craig O'Brien"  wrote on 10/31/2013 01:33:30 PM:

> > Just out of curiosity, why hadn't you already done that?!? 
> 
> I didn't know which host was the problem and didn't think of it. 
> Although I'll readily admit it seems painfully obvious to me now. :)

Just so you're sufficiently humble...  :)

For everyone's future reference:  ALWAYS check the server error log *and* 
the per-host logs...  :)

> >The big question is, though, why they aren't linking.  I'd really 
> start at the bottom of the stack (the physical drives) and work your
> way up.  Check dmesg for any hardware errors.  
> 
> bash-4.1$ grep -i backup /var/log/dmesg*
> bash-4.1$

Nice try, but won't help:  you need to be looking for the correct sd or 
ata device that is used.

Don't bother with a grep like that.  do a dmesg > dmesg.txt and then vi 
(or whatever) dmesg.txt and look for scary errors...  Look particularly 
for sda (or sdb or whatever), or ata0 (or 1 or whatever) messages, or 
possibly scsi messages (yes, SATA is SCSI to Linux) too.

But if they're there, these should not be hard to find:  there tends to be 
*LOTS* of them.

> bash-4.1$ grep -i backup /var/log/messages*

Mine comes back with nothing.

> messages-20131006:Sep 30 13:53:24 servername kernel: BackupPC_dump
> [15365]: segfault at a80 ip 00310f695002 sp 7fff438c9770 
> error 4 in libperl.so[310f60+162000]
> messages-20131006:Sep 30 13:53:27 servername abrtd: Package 
> 'BackupPC' isn't signed with proper key
> messages-20131020:Oct 19 01:24:54 servername kernel: INFO: task 
> BackupPC_dump:11922 blocked for more than 120 seconds.
> messages-20131020:Oct 19 01:24:54 servername kernel: BackupPC_dump D
> 0001 0 11922  10626 0x0080
> messages-20131020:Oct 19 01:30:54 servername kernel: INFO: task 
> BackupPC_dump:11922 blocked for more than 120 seconds.
> messages-20131020:Oct 19 01:30:54 servername kernel: BackupPC_dump D
> 0001 0 11922  10626 0x0080
> messages-20131020:Oct 19 01:32:54 servername kernel: INFO: task 
> BackupPC_dump:11922 blocked for more than 120 seconds.
> messages-20131020:Oct 19 01:32:54 servername kernel: BackupPC_dump D
> 0001 0 11922  10626 0x0080
> messages-20131020:Oct 19 01:32:54 servername kernel: INFO: task 
> BackupPC_nightl:18390 blocked for more than 120 seconds.
> messages-20131020:Oct 19 01:32:54 servername kernel: BackupPC_nigh D
> 0001 0 18390   1262 0x0080
> messages-20131020:Oct 19 01:48:54 servername kernel: INFO: task 
> BackupPC_dump:11922 blocked for more than 120 seconds.
> messages-20131020:Oct 19 01:48:54 servername kernel: BackupPC_dump D
> 0003 0 11922  10626 0x0080
> messages-20131020:Oct 19 01:52:54 servername kernel: INFO: task 
> BackupPC_dump:11922 blocked for more than 120 seconds.
> messages-20131020:Oct 19 01:52:54 servername kernel: BackupPC_dump D
> 0001 0 11922  10626 0x0080
> messages-20131020:Oct 19 01:52:54 servername kernel: INFO: task 
> BackupPC_nightl:18390 blocked for more than 120 seconds.
> messages-20131020:Oct 19 01:52:54 servername kernel: BackupPC_nigh D
> 0001 0 18390   1262 0x0080
> messages-20131020:Oct 19 01:56:54 servername kernel: INFO: task 
> BackupPC_dump:11922 blocked for more than 120 seconds.
> messages-20131020:Oct 19 01:56:54 servername kernel: BackupPC_dump D
> 0003 0 11922  10626 0x0080
> messages-20131020:Oct 19 02:10:54 servername kernel: INFO: task 
> BackupPC_dump:11922 blocked for more than 120 seconds.
> messages-20131020:Oct 19 02:10:54 servername kernel: BackupPC_dump D
> 0001 0 11922  10626 0x0080
> messages-20131020:Oct 19 02:12:54 servername kernel: INFO: task 
> BackupPC_dump:11922 blocked for more than 120 seconds.
> messages-20131020:Oct 19 02:12:54 servername kernel: BackupPC_dump D
> 0001 0 11922  10626 0x0080
> messages-20131027:Oct 23 09:00:02 servername abrtd: Package 
> 'BackupPC' isn't signed with proper key

I'd try Googling those:  they have no meaning for me (and my servers don't 
have them).

What distro are you using?  (I use CentOS/RHEL)

> > fsck the filesystem. 
> 
> bash-4.1$ fsck /dev/sda1
> fsck from util-linux-ng 2.17.2
> e2fsck 1.41.12 (17-May-2010)
> /dev/sda1: clean, 20074506/2929688576 files, 2775975889/2929686016 
blocks
> bash-4.1$

Definitely a good sign.

> >Did I read correctly that this is connected vis NFSv4?  I sure hope
> not...  (I'm willing to admit it's a phobia, but there's no *WAY* I 
> would trust my backup to work across NFS...) 
> 
> The drives are local SATA ones that I set up in a raid 5, directly 
> mounted. Def not NFS. I had an unrelated drive mounted via NFS, but 
> that had nothing to do with my backup system and that's probably the
> source of confusion.

md raid5?  What's the status of /dev/mdstat ?

> So the du command finished, here's the result:
> 
> bash-4.1$ du -hs /backup/pool /backup/cpool /backup/pc/fileserve

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Timothy J Massey]

Les Mikesell  wrote on 10/31/2013 01:54:24 PM:

> On Thu, Oct 31, 2013 at 12:33 PM, Craig O'Brien  
wrote:
> >
> >> fsck the filesystem.
> >
> > bash-4.1$ fsck /dev/sda1
> > fsck from util-linux-ng 2.17.2
> > e2fsck 1.41.12 (17-May-2010)
> > /dev/sda1: clean, 20074506/2929688576 files, 2775975889/2929686016 
blocks
> > bash-4.1$
> 
> That tells you it was unmounted cleanly last time, not that everything
> checks out OK.   Try it with the -f option to make it do the actual
> checks.

Good catch!  This should take a long time:  20 minutes to an hour?  Maybe 
more:  the drives are full.

Tim Massey
 
Out of the Box Solutions, Inc. 
Creative IT Solutions Made Simple!
http://www.OutOfTheBoxSolutions.com
tmas...@obscorp.com 
 
22108 Harper Ave.
St. Clair Shores, MI 48080
Office: (800)750-4OBS (4627)
Cell: (586)945-8796 
--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[backuppc]

Timothy J Massey wrote at about 11:52:35 -0400 on Thursday, October 31, 2013:
 > "Craig O'Brien"  wrote on 10/31/2013 08:49:15 AM:
 > Just out of curiosity, why hadn't you already done that?!?
 > 
 > > Unable to read 8388608 bytes from /var/lib/BackupPC//pc/
 > > myfileserver/new//ffileshare/RStmp got=0, seekPosn=1501757440 (0,
 > > 512,147872,1499463680,2422719488)
 > 
 > Interesting.  I'd make sure that the filesystem is OK before I went much 
 > farther...  Stop BackupPC, unmount /backup and fsck /dev/

Or could be an NFS type error...

 > 
 > > du -hs /backup/pool /backup/cpool /backup/pc/myfileserver/* 
 > > 
 > > to see which backups are doing the most damage. I'll report back 
 > > once that finishes.
 > 
 > With that, you should be able to find the bakup number(s) that are not 
 > linked.  You can delete them and free up space.

Or you could relink the backups to the pool if you want to preserve
the old backups...

 > The big question is, though, why they aren't linking.  I'd really start at 
 > the bottom of the stack (the physical drives) and work your way up.  Check 
 > dmesg for any hardware errors.  fsck the filesystem.  Did I read correctly 
 > that this is connected vis NFSv4?  I sure hope not...  (I'm willing to 
 > admit it's a phobia, but there's no *WAY* I would trust my backup to work 
 > across NFS...)
 
Well I have been using NFS on a NAS for 7 years without
problems... but I probably wouldn't use it in a production, commercial
environment...

--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[backuppc]

Timothy J Massey wrote at about 12:01:06 -0400 on Thursday, October 31, 2013:
 > Holger Parplies  wrote on 10/30/2013 10:24:05 PM:
 > 
 > > as I understand it, the backups from before the change from smb to 
 > rsyncd are
 > > linked into the pool. Since the change, some or all are not. Whether the
 > > change of XferMethod has anything to do with the problem or whether it
 > > coincidentally happened at about the same point in time remains to be 
 > seen.
 > > I still suspect the link to $topDir as cause, and BackupPC_link is 
 > independent
 > > of the XferMethod used (so a change in XferMethod shouldn't have any
 > > influence).
 > 
 > To add my anecdote, I use a symbolic link for all of my BackupPC hosts:  a 
 > couple dozen?  And they all work fine.  It's been my standard procedure 
 > for almost as long as I've been using BackupPC.
 > 
 > Example: 
 > ls -l /var/lib
 > lrwxrwxrwx. 1 rootroot 22 Apr 22  2013 BackupPC -> 
 > /data/BackupPC/TopDir/
 > 
 > mount
 > /dev/sda1 on /data type ext4 (rw)
 > 
 > I understand phobias from earlier problems (see my earlier e-mail about my 
 > thoughts on NFS and backups...) but I do not think this one is an issue.

Agreed - I often link TopDir without any issues...

--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[backuppc]

Craig O'Brien wrote at about 08:49:15 -0400 on Thursday, October 31, 2013:
 > The du -hs /backup/pool /backup/cpool /backup/pc/* has finished. Basically
 > I had 1 host that was taking up 6.9 TB of data with 2.8 TB in the cpool
 > directory and most of the other hosts averaging a GB each.
 > 
 > The 1 host was our file server (which I happen to know has a 2 TB volume
 > (1.3 TB currently used) that is our main fileshare.
 > 
 > I looked through the error log for this pc on backups with the most errors
 > and found thousands of these:
 > 
 > Unable to read 8388608 bytes from
 > /var/lib/BackupPC//pc/myfileserver/new//ffileshare/RStmp got=0,
 > seekPosn=1501757440 (0,512,147872,1499463680,2422719488)
 > Unable to read 8388608 bytes from
 > /var/lib/BackupPC//pc/myfileserver/new//ffileshare/RStmp got=0,
 > seekPosn=1501757440 (0,512,147872,1499463680,2422719488)
 > Unable to read 8388608 bytes from
 > /var/lib/BackupPC//pc/myfileserver/new//ffileshare/RStmp got=0,
 > seekPosn=1501757440 (0,512,147872,1499463680,2422719488)
 > Unable to read 8388608 bytes from
 > /var/lib/BackupPC//pc/myfileserver/new//ffileshare/RStmp got=0,
 > seekPosn=1501757440 (0,512,147872,1499463680,2422719488)
 > Unable to read 8388608 bytes from
 > /var/lib/BackupPC//pc/myfileserver/new//ffileshare/RStmp got=0,
 > seekPosn=1501757440 (0,512,147872,1499463680,2422719488)
 > Unable to read 8388608 bytes from
 > /var/lib/BackupPC//pc/myfileserver/new//ffileshare/RStmp got=0,
 > seekPosn=1501757440 (0,512,147872,1499463680,2422719488)
 > Unable to read 8388608 bytes from
 > /var/lib/BackupPC//pc/myfileserver/new//ffileshare/RStmp got=0,
 > seekPosn=1501757440 (0,512,147872,1499463680,2422719488)
 > Unable to read 8388608 bytes from
 > /var/lib/BackupPC//pc/myfileserver/new//ffileshare/RStmp got=0,
 > seekPosn=1501757440 (0,512,147872,1499463680,2422719488)
 > Unable to read 8388608 bytes from
 > /var/lib/BackupPC//pc/myfileserver/new//ffileshare/RStmp got=0,
 > seekPosn=1501757440 (0,512,147872,1499463680,2422719488)
 > Unable to read 8388608 bytes from
 > /var/lib/BackupPC//pc/myfileserver/new//ffileshare/RStmp got=0,
 > seekPosn=1501757440 (0,512,147872,1499463680,2422719488)
 > Unable to read 8388608 bytes from
 > /var/lib/BackupPC//pc/myfileserver/new//ffileshare/RStmp got=0,
 > seekPosn=1501757440 (0,512,147872,1499463680,2422719488)
 > 

Well, if such errors occur when looking for a matching pool file, then
BackupPC either will create a duplicate pool entry (with a _N pool
chain suffix) or it will fail to link.

So it's possible that either some of your pc files are not linked to
the pool and/or some of the are linked to duplicate (and unnecessary)
pool chains elements...

--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Les Mikesell]

On Thu, Oct 31, 2013 at 12:51 PM, Sharuzzaman Ahmat Raslan
 wrote:
>
> On Fri, Nov 1, 2013 at 1:33 AM, Craig O'Brien  wrote:
>>
>> messages-20131006:Sep 30 13:53:24 servername kernel: BackupPC_dump[15365]:
>> segfault at a80 ip 00310f695002 sp 7fff438c9770 error 4 in
>> libperl.so[310f60+162000]
>
>
> This error shows BackupPC_dump segfault, and pointing to libperl.so
>
> How do you install your BackupPC ? From source or from RPM?
>
> If from RPM, which repo that you use?

Good catch - I missed that line. Bad RAM could cause segfaults too
- and other very mysterious problems.  I had a machine where the
software raid1 mirrors would get differing contents and crash once in
a while - and it took more than a weekend of running memtest to catch
it.

-- 
   Les Mikesell
 lesmikes...@gmail.com

--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[backuppc]

Les Mikesell wrote at about 10:15:42 -0500 on Thursday, October 31, 2013:
 > On Thu, Oct 31, 2013 at 9:47 AM, Craig O'Brien  wrote:
 > >> What is the underlying storage here - nfs?
 > >
 > > Local SATA disks in a RAID 5 (5 disks, 3TB each in capacity)
 > 
 > I think I'd force an fsck just on general principles even though it
 > will take a long time to complete.   Google turns up a few hits on
 > similar problems, but I don't see a definitive answer.   RStmp is
 > supposed to be used to hold an uncompressed copy of the previous
 > version of a large file with changes so rsync can seek to match up the
 > changed block positions, so this error probably has something to do
 > with your compressed copy being corrupted and not uncompressing
 > properly.

And this would explain why the elements are not being linked properly
to the pool -- though I would have thought the more likely result
would be a duplicate pool entry than an unlinked pool entry...

It might be interesting to look for pool chains with the same
(uncompressed) content and with links < HardLinkMax (typically 31999)
to see if pool entries are being unnecessarily duplicated.

Try:
(cd /var/lib/BackupPC/cpool; find . -type f -links -3198 -name "*_*"
-exec md5sum {} \;) | sort | uniq -d -w32

Note this will find if there are any unnecessarily duplicated pool
chains (beyond the base one). Note to keep it fast and simple I am
skipping the elements without a suffix... with the assumption being
that if there are duplicated elements then there will probably be
whole chains of them...

--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Holger Parplies]

Hi,

I've spent far too long writing an email and trying to make it make sense and
then discarding it again.

Just one thought I want to rescue: the RStmp file was really *large*
(something like 1.5 GB), your backup trees are really *large* (1.4 TB), your
pool FS is really *full* (27.5 GB free). Running out of space during a backup
is a bad idea. Both the RStmp file(s) will be truncated (though that should
trigger a second error when it is *written*, just before it is read again) and
the NewFileList, which would, in turn, lead to BackupPC_link missing new files
it would be supposed to link into the pool (resulting in unlinked files).

That doesn't explain your situation, but it still might be something to think
about (and we might be seeing one problem on top of and as result of another).
I agree with Jeffrey - an "Unable to read ..." error *without* a preceeding
"Can't write len=... to .../RStmp" sounds like a mismatch between file length
according to attrib file and result of decompression of compressed file -
probably caused by corruption of the compressed file (or the attrib file,
though unlikely, because the size is not "way off").

How many backups are/were you running in parallel?

Hope that helps.

Regards,
Holger

--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Les Mikesell]

On Thu, Oct 31, 2013 at 2:20 PM, Holger Parplies  wrote:
>>
> That doesn't explain your situation, but it still might be something to think
> about (and we might be seeing one problem on top of and as result of another).
> I agree with Jeffrey - an "Unable to read ..." error *without* a preceeding
> "Can't write len=... to .../RStmp" sounds like a mismatch between file length
> according to attrib file and result of decompression of compressed file -
> probably caused by corruption of the compressed file (or the attrib file,
> though unlikely, because the size is not "way off").

I think that segfault in a perl process needs to be tracked down
before expecting anything else to make sense.  Either bad RAM or
mismatching perl libs could break about anything else.

-- 
   Les Mikesell
lesmikes...@gmail.com

--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Sharuzzaman Ahmat Raslan]

In my experience, segfault in libraries usually caused by installing it
from different source.

For example, when I install BackupPC for CentOS, I use the one in EPEL repo.

I make sure that all the libraries (perl and others), only come from CentOS
base repo, and not from other, as installing them from somewhere else might
cause incompatibilities.

In fact, sometime EPEL repo also provide perl library that conflict with
CentOS base repo, but I just ignore it, and stick to base repo.




On Fri, Nov 1, 2013 at 3:57 AM, Les Mikesell  wrote:

> On Thu, Oct 31, 2013 at 2:20 PM, Holger Parplies 
> wrote:
> >>
> > That doesn't explain your situation, but it still might be something to
> think
> > about (and we might be seeing one problem on top of and as result of
> another).
> > I agree with Jeffrey - an "Unable to read ..." error *without* a
> preceeding
> > "Can't write len=... to .../RStmp" sounds like a mismatch between file
> length
> > according to attrib file and result of decompression of compressed file -
> > probably caused by corruption of the compressed file (or the attrib file,
> > though unlikely, because the size is not "way off").
>
> I think that segfault in a perl process needs to be tracked down
> before expecting anything else to make sense.  Either bad RAM or
> mismatching perl libs could break about anything else.
>
> --
>Les Mikesell
> lesmikes...@gmail.com
>
>
> --
> Android is increasing in popularity, but the open development platform that
> developers love is also attractive to malware creators. Download this white
> paper to learn more about secure code signing practices that can help keep
> Android apps secure.
> http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
> ___
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
>



-- 
Sharuzzaman Ahmat Raslan
--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Craig O'Brien]

> This error shows BackupPC_dump segfault, and pointing to libperl.so
> How do you install your BackupPC ? From source or from RPM?

I did a yum install backuppc, which got it from epel

> That tells you it was unmounted cleanly last time, not that
everything checks out OK.
> Try it with the -f option to make it do the actual checks.

bash-4.1$ fsck -f /dev/sda1
fsck from util-linux-ng 2.17.2
e2fsck 1.41.12 (17-May-2010)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/sda1: 20074505/2929688576 files (0.3% non-contiguous),
2775975116/2929686016 blocks
bash-4.1$

> What distro are you using?  (I use CentOS/RHEL)

CentosOS release 6.4

> How many backups are/were you running in parallel?

Typically 4. But when I switched everything to rsync I wanted fulls done on
all the pc's so I was running up to 8 at a time.

> I think that segfault in a perl process needs to be tracked down before
expecting anything else to make sense.
> Either bad RAM or mismatching perl libs could break about anything else.

I installed perl-libs with yum as well. A yum info perl-libs tells me it
was installed from the updates repo

I think what I'm going to try at this point is to delete the bad backups,
reinstall perl from epel, and keep an eye on it to see if it balloons up
again. Thanks for all your help!


Regards,
Craig


On Thu, Oct 31, 2013 at 10:09 PM, Sharuzzaman Ahmat Raslan <
sharuzza...@gmail.com> wrote:

> In my experience, segfault in libraries usually caused by installing it
> from different source.
>
> For example, when I install BackupPC for CentOS, I use the one in EPEL
> repo.
>
> I make sure that all the libraries (perl and others), only come from
> CentOS base repo, and not from other, as installing them from somewhere
> else might cause incompatibilities.
>
> In fact, sometime EPEL repo also provide perl library that conflict with
> CentOS base repo, but I just ignore it, and stick to base repo.
>
>
>
>
> On Fri, Nov 1, 2013 at 3:57 AM, Les Mikesell wrote:
>
>> On Thu, Oct 31, 2013 at 2:20 PM, Holger Parplies 
>> wrote:
>> >>
>> > That doesn't explain your situation, but it still might be something to
>> think
>> > about (and we might be seeing one problem on top of and as result of
>> another).
>> > I agree with Jeffrey - an "Unable to read ..." error *without* a
>> preceeding
>> > "Can't write len=... to .../RStmp" sounds like a mismatch between file
>> length
>> > according to attrib file and result of decompression of compressed file
>> -
>> > probably caused by corruption of the compressed file (or the attrib
>> file,
>> > though unlikely, because the size is not "way off").
>>
>> I think that segfault in a perl process needs to be tracked down
>> before expecting anything else to make sense.  Either bad RAM or
>> mismatching perl libs could break about anything else.
>>
>> --
>>Les Mikesell
>> lesmikes...@gmail.com
>>
>>
>> --
>> Android is increasing in popularity, but the open development platform
>> that
>> developers love is also attractive to malware creators. Download this
>> white
>> paper to learn more about secure code signing practices that can help keep
>> Android apps secure.
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
>> ___
>> BackupPC-users mailing list
>> BackupPC-users@lists.sourceforge.net
>> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
>> Wiki:http://backuppc.wiki.sourceforge.net
>> Project: http://backuppc.sourceforge.net/
>>
>
>
>
> --
> Sharuzzaman Ahmat Raslan
>
>
> --
> Android is increasing in popularity, but the open development platform that
> developers love is also attractive to malware creators. Download this white
> paper to learn more about secure code signing practices that can help keep
> Android apps secure.
> http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
> ___
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
>
>
--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk___
BackupPC-users mailing list
BackupPC-users@lists.sou

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Craig O'Brien]

>And this would explain why the elements are not being linked properly to
the pool -- though I would have thought the more likely result would be a
duplicate pool entry than an unlinked pool entry...

>It might be interesting to look for pool chains with the same (uncompressed)
content and with links < HardLinkMax (typically 31999) to see if pool
entries are being unnecessarily duplicated.

>Try: (cd /var/lib/BackupPC/cpool; find . -type f -links -3198 -name "*_*" -exec
md5sum {} \;) | sort | uniq -d -w32

> Note this will find if there are any unnecessarily duplicated pool chains
(beyond the base one). Note to keep it fast and simple I am
> skipping the elements without a suffix... with the assumption being that
if there are duplicated elements then there will probably be
> whole chains of them...

bash-4.1$ find . -type f -links -3198 -name "*_*" -exec md5sum {} \; | sort
| uniq -d -w32
71f4cd3f08af68c2ab20c268d86fa9f3  ./c/9/0/c900361b8dc42b2094d836d43504708a_0
bash-4.1$

Looks like this did find something. What should I do with it?

Regards,
Craig


On Fri, Nov 1, 2013 at 9:48 AM, Craig O'Brien  wrote:

> > This error shows BackupPC_dump segfault, and pointing to libperl.so
> > How do you install your BackupPC ? From source or from RPM?
>
> I did a yum install backuppc, which got it from epel
>
> > That tells you it was unmounted cleanly last time, not that
> everything checks out OK.
> > Try it with the -f option to make it do the actual checks.
>
> bash-4.1$ fsck -f /dev/sda1
> fsck from util-linux-ng 2.17.2
> e2fsck 1.41.12 (17-May-2010)
> Pass 1: Checking inodes, blocks, and sizes
> Pass 2: Checking directory structure
> Pass 3: Checking directory connectivity
> Pass 4: Checking reference counts
> Pass 5: Checking group summary information
> /dev/sda1: 20074505/2929688576 files (0.3% non-contiguous),
> 2775975116/2929686016 blocks
> bash-4.1$
>
> > What distro are you using?  (I use CentOS/RHEL)
>
> CentosOS release 6.4
>
> > How many backups are/were you running in parallel?
>
> Typically 4. But when I switched everything to rsync I wanted fulls done
> on all the pc's so I was running up to 8 at a time.
>
> > I think that segfault in a perl process needs to be tracked down before
> expecting anything else to make sense.
> > Either bad RAM or mismatching perl libs could break about anything else.
>
> I installed perl-libs with yum as well. A yum info perl-libs tells me it
> was installed from the updates repo
>
> I think what I'm going to try at this point is to delete the bad backups,
> reinstall perl from epel, and keep an eye on it to see if it balloons up
> again. Thanks for all your help!
>
>
> Regards,
> Craig
>
>
> On Thu, Oct 31, 2013 at 10:09 PM, Sharuzzaman Ahmat Raslan <
> sharuzza...@gmail.com> wrote:
>
>> In my experience, segfault in libraries usually caused by installing it
>> from different source.
>>
>> For example, when I install BackupPC for CentOS, I use the one in EPEL
>> repo.
>>
>> I make sure that all the libraries (perl and others), only come from
>> CentOS base repo, and not from other, as installing them from somewhere
>> else might cause incompatibilities.
>>
>> In fact, sometime EPEL repo also provide perl library that conflict with
>> CentOS base repo, but I just ignore it, and stick to base repo.
>>
>>
>>
>>
>> On Fri, Nov 1, 2013 at 3:57 AM, Les Mikesell wrote:
>>
>>> On Thu, Oct 31, 2013 at 2:20 PM, Holger Parplies 
>>> wrote:
>>> >>
>>> > That doesn't explain your situation, but it still might be something
>>> to think
>>> > about (and we might be seeing one problem on top of and as result of
>>> another).
>>> > I agree with Jeffrey - an "Unable to read ..." error *without* a
>>> preceeding
>>> > "Can't write len=... to .../RStmp" sounds like a mismatch between file
>>> length
>>> > according to attrib file and result of decompression of compressed
>>> file -
>>> > probably caused by corruption of the compressed file (or the attrib
>>> file,
>>> > though unlikely, because the size is not "way off").
>>>
>>> I think that segfault in a perl process needs to be tracked down
>>> before expecting anything else to make sense.  Either bad RAM or
>>> mismatching perl libs could break about anything else.
>>>
>>> --
>>>Les Mikesell
>>> lesmikes...@gmail.com
>>>
>>>
>>> --
>>> Android is increasing in popularity, but the open development platform
>>> that
>>> developers love is also attractive to malware creators. Download this
>>> white
>>> paper to learn more about secure code signing practices that can help
>>> keep
>>> Android apps secure.
>>>
>>> http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
>>> ___
>>> BackupPC-users mailing list
>>> BackupPC-users@lists.sourceforge.net
>>> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
>>> Wiki:http://backuppc.wiki.sourceforge.net
>>> Project: http://bac

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Les Mikesell]

On Fri, Nov 1, 2013 at 8:48 AM, Craig O'Brien  wrote:
>> This error shows BackupPC_dump segfault, and pointing to libperl.so
>> How do you install your BackupPC ? From source or from RPM?
>
> I did a yum install backuppc, which got it from epel

Do you see any other segfaults in your logs (not necessarily just from
backuppc)?

>> How many backups are/were you running in parallel?
>
> Typically 4. But when I switched everything to rsync I wanted fulls done on
> all the pc's so I was running up to 8 at a time.

Most machines would get better overall throughput with a max of 2
concurrent runs (depending on a lot of things, of course...).

>> I think that segfault in a perl process needs to be tracked down before
>> expecting anything else to make sense.
>> Either bad RAM or mismatching perl libs could break about anything else.
>
> I installed perl-libs with yum as well. A yum info perl-libs tells me it was
> installed from the updates repo

Have you installed anything from repos other than the CentOS base and
EPEL?  You shouldn't have any trouble with anything from those.

-- 
   Les Mikesell
  lesmikes...@gmail.com

--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Timothy J Massey]

"Craig O'Brien"  wrote on 11/01/2013 09:48:23 AM:

> > This error shows BackupPC_dump segfault, and pointing to libperl.so
> > How do you install your BackupPC ? From source or from RPM?
> 
> I did a yum install backuppc, which got it from epel

That's how I do it.

> > That tells you it was unmounted cleanly last time, not that 
> everything checks out OK.   
> > Try it with the -f option to make it do the actual checks.
> 
> bash-4.1$ fsck -f /dev/sda1
> fsck from util-linux-ng 2.17.2
> e2fsck 1.41.12 (17-May-2010)
> Pass 1: Checking inodes, blocks, and sizes
> Pass 2: Checking directory structure
> Pass 3: Checking directory connectivity
> Pass 4: Checking reference counts
> Pass 5: Checking group summary information
> /dev/sda1: 20074505/2929688576 files (0.3% non-contiguous), 
> 2775975116/2929686016 blocks
> bash-4.1$

Good.  I think we've eliminated a disk or filesystem issue.  I think we're 
pretty comfortable it's a BackupPC corruption issue.  It was hard to tell 
when your error messages said that it could not seek to a particular point 
in a file.

> > What distro are you using?  (I use CentOS/RHEL) 
> 
> CentosOS release 6.4

Same here.

> > I think that segfault in a perl process needs to be tracked 
> down before expecting anything else to make sense.  
> > Either bad RAM or mismatching perl libs could break about anything 
else.
> 
> I installed perl-libs with yum as well. A yum info perl-libs tells 
> me it was installed from the updates repo
> 
> I think what I'm going to try at this point is to delete the bad 
> backups, reinstall perl from epel, and keep an eye on it to see if 
> it balloons up again. Thanks for all your help!

That's a very reasonable, if not very subtle, solution.

I think you need to monitor /var/log/messages for errors that mention 
backup.  See if the crash returns.  Jeff is (justifiably) worried that the 
crash caused your corruption, but it could just as easily be the other way 
around.  Once you clean up from this, you want to make sure that nothing 
comes back.

If you've got the time, running memtest for a weekend might be a good 
idea, too.  The only thing it would cost is the downtime...

Tim Massey
 
Out of the Box Solutions, Inc. 
Creative IT Solutions Made Simple!
http://www.OutOfTheBoxSolutions.com
tmas...@obscorp.com 
 
22108 Harper Ave.
St. Clair Shores, MI 48080
Office: (800)750-4OBS (4627)
Cell: (586)945-8796 
--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[backuppc]

Craig O'Brien wrote at about 10:11:07 -0400 on Friday, November 1, 2013:
 > >And this would explain why the elements are not being linked properly to
 > the pool -- though I would have thought the more likely result would be a
 > duplicate pool entry than an unlinked pool entry...
 > 
 > >It might be interesting to look for pool chains with the same (uncompressed)
 > content and with links < HardLinkMax (typically 31999) to see if pool
 > entries are being unnecessarily duplicated.
 > 
 > >Try: (cd /var/lib/BackupPC/cpool; find . -type f -links -3198 -name "*_*" 
 > >-exec
 > md5sum {} \;) | sort | uniq -d -w32
 > 
 > > Note this will find if there are any unnecessarily duplicated pool chains
 > (beyond the base one). Note to keep it fast and simple I am
 > > skipping the elements without a suffix... with the assumption being that
 > if there are duplicated elements then there will probably be
 > > whole chains of them...
 > 

I added some more bash-foo so that the following should find *any* and *all*
unnecessary pool dups...

(cd /var/lib/BackupPC/cpool; find . -name "*_0" | sed "s/_0$//" | (IFS=$'\n'; 
while read FILE; do find "${FILE}"* -links -3199 -exec md5sum {} \; | sort | 
uniq -D -w32 ; done))

Then do an 'ls -ial' to find their size and number of links they each
have. The "-i" will also tell you the inode for later reference.

--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Holger Parplies]

Hi,

I get some diagnostics when reading this with 'use warnings "wrong_numbers"' ...

backu...@kosowsky.org wrote on 2013-11-01 12:18:17 -0400 [Re: [BackupPC-users] 
Disk space used far higher than reported pool?size]:
> Craig O'Brien wrote at about 10:11:07 -0400 on Friday, November 1, 2013:
>  > >And this would explain why the elements are not being linked properly to
>  > the pool -- though I would have thought the more likely result would be a
>  > duplicate pool entry than an unlinked pool entry...
>  > 
>  > >It might be interesting to look for pool chains with the same 
> (uncompressed)
>  > content and with links < HardLinkMax (typically 31999) to see if pool

this one looks correct. 31999. Unless of course you've changed it in config.pl
because your FS requirements differ.

>  > entries are being unnecessarily duplicated.
>  > 
>  > >Try: (cd /var/lib/BackupPC/cpool; find . -type f -links -3198 -name "*_*" 
> -exec

This one doesn't.

>  > md5sum {} \;) | sort | uniq -d -w32
>  > 
>  > > Note this will find if there are any unnecessarily duplicated pool chains
>  > (beyond the base one). Note to keep it fast and simple I am
>  > > skipping the elements without a suffix... with the assumption being that
>  > if there are duplicated elements then there will probably be
>  > > whole chains of them...
>  > 
> 
> I added some more bash-foo so that the following should find *any* and *all*
> unnecessary pool dups...
> 
> (cd /var/lib/BackupPC/cpool; find . -name "*_0" | sed "s/_0$//" | (IFS=$'\n'; 
> while read FILE; do find "${FILE}"* -links -3199 -exec md5sum {} \; | sort | 
> uniq -D -w32 ; done))

Nor does this one (the 3199 again). While it will find chain members with
less links than apparently necessary, it won't find all of them - only those
with *far* too small link number. That might be sufficient, depending on what
we're looking for. You probably wouldn't have chosen the (arbitrary) value
"3199", though, if you hadn't in fact meant "31999" ;-). And you wouldn't be
saying "*any* and *all*" if you were meaning "some".

I'd like to point out three things:
1.) unnecessary duplication *within* the pool is not the problem we are
looking for,
2.) if it were a problem, then because a duplicate was created way ahead of
time and repeatedly, not because the overflow happens at 31950 instead of
31999,
3.) finding "unnecessary duplicates" can have a normal explanation: if at some
point you had more than 31999 copies of one file (content) in your
backups, BackupPC would have created a pool duplicate. Some of the backups
linking to the first copy would have expired over time, leaving behind a
link count < 31999. Further rsync backups would tend to link to the second
copy, at least for unchanging existing files (in full backups). In other
cases, the first copy might be reused, but there's no guarantee the link
count would be exactly 31999 (though it would probably tend to be).
Having so many copies of identical file content in your backups would tend
to happen for small files rather than huge ones, I would expect, and it
doesn't seem to be very common anyway (in my pools, I find exactly one file
with a link count of 60673 (XFS) and a total of five with more than 1
links, the largest having 103 bytes (compressed)).

Regards,
Holger

--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[backuppc]

Holger Parplies wrote at about 18:57:05 +0100 on Friday, November 1, 2013:
 > Hi,
 > 
 > I get some diagnostics when reading this with 'use warnings "wrong_numbers"' 
 > ...
 > 
 > backu...@kosowsky.org wrote on 2013-11-01 12:18:17 -0400 [Re: 
 > [BackupPC-users] Disk space used far higher than reported pool?size]:
 > > Craig O'Brien wrote at about 10:11:07 -0400 on Friday, November 1, 2013:
 > >  > >And this would explain why the elements are not being linked properly 
 > > to
 > >  > the pool -- though I would have thought the more likely result would be 
 > > a
 > >  > duplicate pool entry than an unlinked pool entry...
 > >  > 
 > >  > >It might be interesting to look for pool chains with the same 
 > > (uncompressed)
 > >  > content and with links < HardLinkMax (typically 31999) to see if pool
 > 
 > this one looks correct. 31999. Unless of course you've changed it in 
 > config.pl
 > because your FS requirements differ.
 > 
 > >  > entries are being unnecessarily duplicated.
 > >  > 
 > >  > >Try: (cd /var/lib/BackupPC/cpool; find . -type f -links -3198 -name 
 > > "*_*" -exec
 > 
 > This one doesn't.

Oops typo... dropped a 9
 > 
 > >  > md5sum {} \;) | sort | uniq -d -w32
 > >  > 
 > >  > > Note this will find if there are any unnecessarily duplicated pool 
 > > chains
 > >  > (beyond the base one). Note to keep it fast and simple I am
 > >  > > skipping the elements without a suffix... with the assumption being 
 > > that
 > >  > if there are duplicated elements then there will probably be
 > >  > > whole chains of them...
 > >  > 
 > > 
 > > I added some more bash-foo so that the following should find *any* and 
 > > *all*
 > > unnecessary pool dups...
 > > 
 > > (cd /var/lib/BackupPC/cpool; find . -name "*_0" | sed "s/_0$//" | 
 > > (IFS=$'\n'; while read FILE; do find "${FILE}"* -links -3199 -exec md5sum 
 > > {} \; | sort | uniq -D -w32 ; done))
 > 
 > Nor does this one (the 3199 again). 
Typo again... dropped a 9

 > You probably wouldn't have chosen the (arbitrary) value
 > "3199", though, if you hadn't in fact meant "31999" ;-). And you wouldn't be
 > saying "*any* and *all*" if you were meaning "some".
 > 
 > I'd like to point out three things:
 > 1.) unnecessary duplication *within* the pool is not the problem we are
 > looking for,

This is probably not his *primary* issue since the pool is (only)
~3T. But when he started talking about file read errors, I was
concerned that if the pool file reads were being truncated, then there
likely would be pool duplicates since the byte-by-byte comparisons
would fail for a given partial file md5sum leading to extra chain creation...

 > 2.) if it were a problem, then because a duplicate was created way ahead of
 > time and repeatedly, not because the overflow happens at 31950 instead of
 > 31999,

 > 3.) finding "unnecessary duplicates" can have a normal explanation: if at 
 > some
 > point you had more than 31999 copies of one file (content) in your
 > backups, BackupPC would have created a pool duplicate. Some of the 
 > backups
 > linking to the first copy would have expired over time, leaving behind a
 > link count < 31999. Further rsync backups would tend to link to the 
 > second
 > copy, at least for unchanging existing files (in full backups). In other
 > cases, the first copy might be reused, but there's no guarantee the link
 > count would be exactly 31999 (though it would probably tend to
 > be).


You are absolutely right that there are valid reasons for the link
count overflowing 31999 and then later dropping below as links are
expired.

To tell you the truth, my use of "-links 31999" (corrected) was really
more pedantic -- in reality, I have never seen a case of link counts
getting that high... and if it does, it's probably extremely rare to
have a single non-zero file repeated that many times unless you have a
huge set of clients or a huge set of past full backups... (or some
special situation where users keep large numbers of copies of certain
files).

So, basically, while there may be an exceptional case or two, anything
spewed back by my shell one-liner is worth looking at from the
perspective of potential issues with pool duplication.

 > Having so many copies of identical file content in your backups would 
 > tend
 > to happen for small files rather than huge ones, I would expect, and it
 >

Re: [BackupPC-users] Disk space used far higher than reported pool size

[backuppc]

Holger Parplies wrote at about 18:57:05 +0100 on Friday, November 1, 2013:
 > 3.) finding "unnecessary duplicates" can have a normal explanation: if at 
 > some
 > point you had more than 31999 copies of one file (content) in your
 > backups, BackupPC would have created a pool duplicate. Some of the 
 > backups
 > linking to the first copy would have expired over time, leaving behind a
 > link count < 31999. Further rsync backups would tend to link to the 
 > second
 > copy, at least for unchanging existing files (in full backups). In other
 > cases, the first copy might be reused, but there's no guarantee the link
 > count would be exactly 31999 (though it would probably tend to be).

Interesting... I think this depends on the transfer method. The rsync
method looks to the immediately prior full for comparison, so new hard
links will be made to the same chain as the last full. Thus, if
earlier elements in the chain have reduced link count, they will tend
not to be filled back in.

It seems like the other transfer methods directly reference the
PoolWrite package which always crawls up the chain looking for
matches... If true, it does seem that one could in general, speed up
fulls for the other algorithms by putting a matching candidate from
the previous full (if any) first in the candidate match list... rather
than matching them in chain order (or several simultaneously).

In any case, if my quick reading of the code is correct, then the
other methods will tend to fill in earlier chains elements first so
that the link count will march back up to 31999.

--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Disk space used far higher than reported pool size

[Les Mikesell]

On Fri, Nov 1, 2013 at 1:46 PM,   wrote:
>
> This is probably not his *primary* issue since the pool is (only)
> ~3T. But when he started talking about file read errors, I was
> concerned that if the pool file reads were being truncated, then there
> likely would be pool duplicates since the byte-by-byte comparisons
> would fail for a given partial file md5sum leading to extra chain creation...

The read errors were in the RStmp files that is supposed to be the
uncompressed copy of a large compressed file so rsync can seek around
looking for a match.   I wonder if there could be a file (huge
database, mailbox,etc.) that compresses to a point that even with the
safety factor of backups not starting at 95% full, the uncompressed
copy won't fit.   Or maybe a sparse dbm type file where the original
doesn't allocate the space the length would indicate.

-- 
   Les Mikesell
  lesmikes...@gmail.com

--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/