The partition where the "pc" folder lives needs to be larger than the size of
your largest backup target. It will grow to the size of whatever is being
backed up at that moment. If you have simultaneous backups, I believe it could
grow larger than your largest backup target.
Here is how I came to that conclusion: I have been monitoring changes in the
size of the "pc" folder every second to learn how to anticipate disk usage and
build a better server. These are the highs and lows experienced over a typical
evening:
01:00:29 0G
01:04:30 2G
01:07:57 1G
01:11:36 6G
01:22:16 0G
01:27:13 1G
01:31:41 0G
01:53:22 25G
02:12:39 0G
02:31:10 23G
03:05:59 0G
This was measured while backing up seven RHEL web/mysql servers on a LAN; some
with large database dumps. All backups triggered at 1:00a and most completed in
20 minutes. The other two servers performed full backups and took about two
hours each, and must be responsible for the 20GB spikes. I can confirm by
looking at the "File Size/Count Reuse Summary" table that their sizes are
20-25GB. Add to that the potential of simultaneous runs so be sure your "pc"
folder is on a large enough partition.
From: Mike Hughes
Sent: Friday, August 24, 2018 09:56
To: backuppc-users@lists.sourceforge.net
Subject: RE: [BackupPC-users] ver 4.x split using ssd and hdd storage - size
requirements?
>-----Original Message-----
>From: Johan Ehnberg <jo...@molnix.com<mailto:jo...@molnix.com>>
>Sent: Friday, August 24, 2018 09:25
>To:
>backuppc-users@lists.sourceforge.net<mailto:backuppc-users@lists.sourceforge.net>
>Subject: Re: [BackupPC-users] ver 4.x split using ssd and hdd storage - size
>requirements?
>
>On 08/24/2018 04:52 PM, Mike Hughes wrote:
>> I think I've discovered a new level of failure.
...
>> Can't write new host pool count file
>> /var/lib/BackupPC//pc/hostname/refCnt/poolCntNew.1.02
>>
>> My guess is that the /var/lib/BackupPC/pc partition is the problem.
...
>> https://molnix.com/backuppc-version-4-development-allows-better-scaling/
>>
>
>Hi Mike,
>
>Post author here, nice to hear it is of use!
Great! Nice to see you again John!
>Can you check the size of the files that trigger the errors? What does
>df -h tell you during backups? If the failed refcount files linger on
>your drive, how big do they grow?
That was a really good idea and I think you nailed it. Here's a one-second
update on used space for the /var partition during an incremental backup:
[Cent-7:root@hostname ~]# while printf '%s ' "$(df -P /var | awk 'NR==2 { print
$(NF-1) }')"; do sleep 1; done
28% 28% 28% 28% 28% 28% 28% 29% 29% 31% 31% 32% 34% 34% 35% 35% 35% 35% 35% 35%
36% 36% 36% 36% 36% 36% 36% 36% 36% 36% 36% 36% 38% 40% 42% 43% 43% 43% 44% 44%
44% 44% 44% 45% 45% 45% 45% 45% 45% 45% 45% 45% 45% 45% 46% 46% 46% 46% 46% 46%
46% 49% 52% 54% 54% 54% 54% 54% 54% 54% 54% 54% 54% 54% 54% 54% 54% 54% 54% 54%
54% 57% 59% 61% 61% 61% 62% 62% 62% 62% 62% 62% 62% 63% 63% 63% 63% 63% 63% 63%
63% 63% 63% 63% 63% 63% 63% 63% 63% 63% 63% 64% 67% 69% 70% 70% 71% 71% 71% 71%
71% 71% 71% 72% 72% 72% 72% 72% 72% 72% 72% 72% 72% 72% 72% 72% 72% 72% 72% 72%
72% 72% 72% 72% 76% 79% 80% 80% 80% 80% 80% 81% 81% 81% 81% 81% 81% 81% 81% 81%
81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81%
81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81%
81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81%
81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81%
81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81%
81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81%
81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81%
81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 29% 31% 31% 32% 33% 36% 36%
36% 36% 37% 37% 37% 37% 37% 37% 37% 37% 38% 38% 38% 38% 38% 38% 38% 38% 39% 39%
39% 39% 39% 39% 39% 39% 39% 39% 39% 39% 39% 39% 39% 41% 42% 43% 43% 44% 46% 48%
48% 48% 49% 49% 49% 49% 49% 49% 49% 49% 49% 49% 49% 49% 49% 49% 49% 49% 49% 49%
50% 53% 56% 57% 57% 57% 58% 58% 58% 58% 58% 58% 58% 59% 59% 59% 59% 59% 59% 59%
59% 59% 59% 59% 59% 59% 59% 59% 59% 59% 59% 59% 59% 59% 60% 63% 65% 67% 67% 68%
68% 68% 68% 68% 69% 69% 69% 69% 69% 69% 69% 70% 70% 70% 70% 70% 71% 71% 71% 71%
71% 71% 71% 71% 71% 71% 71% 71% 71% 71% 71% 71% 71% 71% 71% 71% 71% 72% 76% 79%
81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81% 81%
84% 86% 89% 89% 89% 90% 90% 90% 91% 91% 91% 91% 91% 91% 91% 91% 91% 91% 91% 91%
91% 91% 91% 91% 91% 91% 92% 92% 92% 92% 92% 92% 92% 92% 92% 94% 96% 98% 99%
100% 100% 100% 100% 100% 100% 100% 93% 93% 28% 28%
If you're able to receive images:
[cid:image001.png@01D43B8F.5C49A780]
caption: NewRelic showing close to but not reaching 100% utilization
So it's definitely requiring more storage than I expected. I love the idea of
using an SSD to increase performance but I need to better understand the growth
potential during backups, especially during simultaneous backups.
>I will follow this thread with interest and hopefully I have the time to
>closely monitor what happens in the pc folder during reference counts.
>
>Best regards,
>Johan
>
>--
>Johan Ehnberg
>Founder, CEO
>jo...@molnix.com<mailto:jo...@molnix.com>
>+358503209688
>
>Molnix Oy
>molnix.com
Thank you!
Mike
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
