Re: [BackupPC-users] How does BackupPC 4.1.1 calculate Pools size?

2017-05-13 Thread Craig Barratt 
I just pushed a change

to BackupPC_nightly that removes any orphan temp write pool files.

Craig

On Sun, May 7, 2017 at 1:17 AM, Craig Barratt <
cbarr...@users.sourceforge.net> wrote:

> Yes, you're right.
>
> Looks like they got left over from an abnormal backup exit.  I need to add
> some code that cleans up any that remain after the backup.
>
> It's certainly safe to remove them (assuming they are hours/days old).  If
> you want to be really sure, the first integer is the pid of the process
> that created them, in this case 19524.  You could confirm that process no
> longer exists before deleting them.
>
> Craig
>
>
> On Friday, May 5, 2017, Michael Huntley  wrote:
>
>> Based upon my observations when testing it appears these are temporary
>> files prior to being added to the pool.
>>
>>
>> Craig - any input?
>>
>> Cheers!
>>
>> mph
>>
>> On 2017-05-05 10:29, Dieter Fauth wrote:
>>
>> Hi Ray,
>> I have similar files in cpool. ~60GB in 6 files.
>> I suspected they were created in my experiments with archive host. At
>> least the file date would fit.
>> But that was a wild guess.
>>
>> Anybody else seeing such files?
>>
>> On Fri, 05 May 2017 18:15:23 +0200, Ray Frush 
>>
>> wrote:
>>
>> I spotted two file stored in the cpool totaling 23.5GB, which is about
>> the
>> same as the discrepancy between the pool size that BackupPC reports and
>> what I'm counting on the file system.
>>
>> ...
>> drwxr-x---. 130 backuppc backuppc 8.0K May  4 01:00 18
>> -rw-rw.   1 backuppc backuppc  16G Apr 27 14:38 19524.1398.0
>> -rw-rw.   1 backuppc backuppc 7.5G Apr 27 14:44 19524.1399.0
>> drwxr-x---. 130 backuppc backuppc 8.0K May  4 01:00 1a
>> drwxr-x---. 130 backuppc backuppc 8.0K May  4 01:00 1c
>> ...
>>
>> Can you help me understand what these files are for?
>>
>>
>>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] How does BackupPC 4.1.1 calculate Pools size?

2017-05-07 Thread Craig Barratt 
Yes, you're right.

Looks like they got left over from an abnormal backup exit.  I need to add
some code that cleans up any that remain after the backup.

It's certainly safe to remove them (assuming they are hours/days old).  If
you want to be really sure, the first integer is the pid of the process
that created them, in this case 19524.  You could confirm that process no
longer exists before deleting them.

Craig

On Friday, May 5, 2017, Michael Huntley  wrote:

> Based upon my observations when testing it appears these are temporary
> files prior to being added to the pool.
>
>
> Craig - any input?
>
> Cheers!
>
> mph
>
> On 2017-05-05 10:29, Dieter Fauth wrote:
>
> Hi Ray,
> I have similar files in cpool. ~60GB in 6 files.
> I suspected they were created in my experiments with archive host. At
> least the file date would fit.
> But that was a wild guess.
>
> Anybody else seeing such files?
>
> On Fri, 05 May 2017 18:15:23 +0200, Ray Frush  >
> wrote:
>
> I spotted two file stored in the cpool totaling 23.5GB, which is about
> the
> same as the discrepancy between the pool size that BackupPC reports and
> what I'm counting on the file system.
>
> ...
> drwxr-x---. 130 backuppc backuppc 8.0K May  4 01:00 18
> -rw-rw.   1 backuppc backuppc  16G Apr 27 14:38 19524.1398.0
> -rw-rw.   1 backuppc backuppc 7.5G Apr 27 14:44 19524.1399.0
> drwxr-x---. 130 backuppc backuppc 8.0K May  4 01:00 1a
> drwxr-x---. 130 backuppc backuppc 8.0K May  4 01:00 1c
> ...
>
> Can you help me understand what these files are for?
>
>
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] How does BackupPC 4.1.1 calculate Pools size?

2017-05-05 Thread Michael Huntley 
Based upon my observations when testing it appears these are temporary
files prior to being added to the pool.

Craig - any input? 

Cheers! 

mph 

On 2017-05-05 10:29, Dieter Fauth wrote:

> Hi Ray,
> I have similar files in cpool. ~60GB in 6 files.
> I suspected they were created in my experiments with archive host. At  
> least the file date would fit.
> But that was a wild guess.
> 
> Anybody else seeing such files?
> 
> On Fri, 05 May 2017 18:15:23 +0200, Ray Frush   
> wrote:
> 
>> I spotted two file stored in the cpool totaling 23.5GB, which is about  
>> the
>> same as the discrepancy between the pool size that BackupPC reports and
>> what I'm counting on the file system.
>> 
>> ...
>> drwxr-x---. 130 backuppc backuppc 8.0K May  4 01:00 18
>> -rw-rw.   1 backuppc backuppc  16G Apr 27 14:38 19524.1398.0
>> -rw-rw.   1 backuppc backuppc 7.5G Apr 27 14:44 19524.1399.0
>> drwxr-x---. 130 backuppc backuppc 8.0K May  4 01:00 1a
>> drwxr-x---. 130 backuppc backuppc 8.0K May  4 01:00 1c
>> ...
>> 
>> Can you help me understand what these files are for?--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] How does BackupPC 4.1.1 calculate Pools size?

2017-05-05 Thread Dieter Fauth 
Hi Ray,
I have similar files in cpool. ~60GB in 6 files.
I suspected they were created in my experiments with archive host. At  
least the file date would fit.
But that was a wild guess.

Anybody else seeing such files?

On Fri, 05 May 2017 18:15:23 +0200, Ray Frush   
wrote:

> I spotted two file stored in the cpool totaling 23.5GB, which is about  
> the
> same as the discrepancy between the pool size that BackupPC reports and
> what I'm counting on the file system.
>
> ...
> drwxr-x---. 130 backuppc backuppc 8.0K May  4 01:00 18
> -rw-rw.   1 backuppc backuppc  16G Apr 27 14:38 19524.1398.0
> -rw-rw.   1 backuppc backuppc 7.5G Apr 27 14:44 19524.1399.0
> drwxr-x---. 130 backuppc backuppc 8.0K May  4 01:00 1a
> drwxr-x---. 130 backuppc backuppc 8.0K May  4 01:00 1c
> ...
>
> Can you help me understand what these files are for?
>
>
>


-- 
Many regards, Dieter Fauth :-)

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] How does BackupPC 4.1.1 calculate Pools size?

2017-05-05 Thread Ray Frush 
I spotted two file stored in the cpool totaling 23.5GB, which is about the
same as the discrepancy between the pool size that BackupPC reports and
what I'm counting on the file system.

...
drwxr-x---. 130 backuppc backuppc 8.0K May  4 01:00 18
-rw-rw.   1 backuppc backuppc  16G Apr 27 14:38 19524.1398.0
-rw-rw.   1 backuppc backuppc 7.5G Apr 27 14:44 19524.1399.0
drwxr-x---. 130 backuppc backuppc 8.0K May  4 01:00 1a
drwxr-x---. 130 backuppc backuppc 8.0K May  4 01:00 1c
...

Can you help me understand what these files are for?



-- 
Time flies like an arrow, but fruit flies like a banana.
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] How does BackupPC 4.1.1 calculate Pools size?

2017-05-04 Thread Ray Frush 
Craig-
I set "$Conf{PoolSizeNightlyUpdatePeriod} = 1;" yesterday, and really
didn't see a change in the size discrepancy between what BackupPC reports
and the file system still remains, after letting it run with the updated
config for two nights.  The estimate from my last message grew from 26GiB
to 29GiB, while the actual space used on the disk grew only 1GB.   Is is
possible that the process is still using the default value of 16 anyway?


The WebUI shows: "Pool is 29.08GiB comprising 779885 files and 16512
directories (as of 5/4 01:00),"

And a check of the base file system shows:
# du -bhs *
52G cpool
2.2G pc
4.0K pool


My log output from last night suggests that the Nightly job really didn't
take very long, and I question if it had time to scan the pool for actual
size used:

2017-05-04 01:00:00 24hr disk usage: 5% max, 5% recent, 0 skipped hosts
2017-05-04 01:00:00 Aging LOG files, LOG -> LOG.0 -> LOG.1 -> ... -> LOG.13
...
2017-05-04 01:00:00 Running 2 BackupPC_nightly jobs from 0..15 (out of
0..15)
2017-05-04 01:00:00 Running BackupPC_nightly -m -P 6 0 127 (pid=21588)
2017-05-04 01:00:00 Running BackupPC_nightly -P 6 128 255 (pid=21589)
2017-05-04 01:00:00 Next wakeup is 2017-05-04 02:00:00
2017-05-04 01:00:01 BackupPC_nightly now running BackupPC_refCountUpdate -m
-s -c -P 6 -r 0-127
2017-05-04 01:00:01 BackupPC_nightly now running BackupPC_refCountUpdate -m
-s -c -P 6 -r 128-255
2017-05-04 01:00:01  admin1 : __bpc_pidStart__ 21604
2017-05-04 01:00:01  admin : __bpc_pidStart__ 21603
2017-05-04 01:00:36  admin : __bpc_pidEnd__ 21603
2017-05-04 01:00:36 BackupPC_nightly now running BackupPC_sendEmail
2017-05-04 01:00:37  admin1 : __bpc_pidEnd__ 21604
2017-05-04 01:00:37 Finished  admin1  (BackupPC_nightly -P 6 128 255)
2017-05-04 01:00:40 Finished  admin  (BackupPC_nightly -m -P 6 0 127)
2017-05-04 01:00:40 Pool nightly clean removed 0 files of size 0.00GB
2017-05-04 01:00:40 Pool is 0.00GB, 0 files (0 repeated, 0 max chain, 0 max
links), 0 directories
2017-05-04 01:00:40 Cpool nightly clean removed 0 files of size 0.00GB
2017-05-04 01:00:40 Cpool is 0.00GB, 0 files (0 repeated, 0 max chain, 0
max links), 0 directories
2017-05-04 01:00:40 Pool4 nightly clean removed 0 files of size 0.00GB
2017-05-04 01:00:40 Pool4 is 0.00GB, 0 files (0 repeated, 0 max chain, 0
max links), 0 directories
2017-05-04 01:00:40 Cpool4 nightly clean removed 603 files of size 0.06GB
2017-05-04 01:00:40 Cpool4 is 29.77GB, 779885 files (0 repeated, 0 max
chain, 4334 max links), 16512 directories
2017-05-04 01:00:40 Running BackupPC_rrdUpdate (pid=21628)
2017-05-04 01:00:43  admin-1 : 2017-05-04 01:00:43 RRD updated: date
1493942400; cpoolKb 0.00; total 719841210.985352; poolKb 0.00;
pool4Kb 0.00; cpool4Kb 30488312.00
2017-05-04 01:00:45 Finished  admin-1  (BackupPC_rrdUpdate)


Thanks again for taking a look at this.



On Mon, May 1, 2017 at 10:50 PM, Craig Barratt  wrote:

> The nightly pool check (BackupPC_nightly) only traverses a portion of the
> pool each night.  See $Conf{PoolSizeNightlyUpdatePeriod}.  The default is
> 16, meaning it takes 16 nightly runs to get through the whole pool.
>
> It looks like your installation is quite small, so you could
> set $Conf{PoolSizeNightlyUpdatePeriod} to 1.
>
> Craig
>
> On Mon, May 1, 2017 at 3:48 PM, Ray Frush 
> wrote:
>
>> My instance of Backuppc 4.1.1 reports:
>>
>> "Pool is 26.56GiB comprising 764047 files and 16512 directories (as of
>> 5/1 01:00)"
>>
>> However, when I check the file system, I get 'slightly' different numbers:
>>
>> # du -bhs *
>> 50G cpool
>> 2.1G pc
>> 4.0K pool
>>
>>
>> 26GiB  vs ~52GB.
>>
>> The target file system is an NFS file system.   Is there something I
>> should be doing different to get a more accurate report of the pool size?
>> How does BackupPC calculate the pool size?   (I'm trying to grok the source
>> code, but haven't found the method yet.)
>>
>>
>> Thanks
>>
>>
>> --
>> Ray Frush
>> Colorado State University
>>
>>

-- 
Time flies like an arrow, but fruit flies like a banana.
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] How does BackupPC 4.1.1 calculate Pools size?

2017-05-01 Thread Craig Barratt 
The nightly pool check (BackupPC_nightly) only traverses a portion of the
pool each night.  See $Conf{PoolSizeNightlyUpdatePeriod}.  The default is
16, meaning it takes 16 nightly runs to get through the whole pool.

It looks like your installation is quite small, so you could
set $Conf{PoolSizeNightlyUpdatePeriod} to 1.

Craig

On Mon, May 1, 2017 at 3:48 PM, Ray Frush  wrote:

> My instance of Backuppc 4.1.1 reports:
>
> "Pool is 26.56GiB comprising 764047 files and 16512 directories (as of 5/1
> 01:00)"
>
> However, when I check the file system, I get 'slightly' different numbers:
>
> # du -bhs *
> 50G cpool
> 2.1G pc
> 4.0K pool
>
>
> 26GiB  vs ~52GB.
>
> The target file system is an NFS file system.   Is there something I
> should be doing different to get a more accurate report of the pool size?
> How does BackupPC calculate the pool size?   (I'm trying to grok the source
> code, but haven't found the method yet.)
>
>
> Thanks
>
>
> --
> Ray Frush
> Colorado State University
>
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
>
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/