Re: [BackupPC-users] Backuppc 3.3.0 - Full backups not backing up everything
Thanks you for your reply. The problem is with windows clients and we were backing these clients up with 3.1.0 with out experiencing this problem. It is more than one or two files not being backed up, it is that backuppc does not appear to even attempt backing up major directories such as Users so I doubt locked files are the problem. Thanks again. On 10/24/2013 8:47 PM, Prem wrote: Based on my experinced...if Windows could be due to open files.. On linux, possibly due to permission where the user account used (other than root) does not have permission to access the directories. Update your sudoers file for this. From: Dean Lambourn de...@pcf.com To: backuppc-users@lists.sourceforge.net Sent: Friday, October 25, 2013 12:48 AM Subject: [BackupPC-users] Backuppc 3.3.0 - Full backups not backing up everything Recently installed backuppc 3.3.0 on a new server running Debian Jessie. Full backups are not backing up all the directories. Initially server was configured with the BackupFilesOnly variable set to \users and a couple of other directories. When that failed, I deleted the variable and attempted to back up everything which was also unsuccessful. Incremental backups seem to at least get the entire directory tree but can't tell if it is backing up all the files probably because I ran the incremental right after checking the full backup and nothing changed in the directories I checked. The backup is using smb and I've attached the config.pl for the server in question as well as the Xferlog and backupInfo from the full backup. I have also included a PDF with screen shots of the directory tree for the test client, the web page listing of the full backup and the web page listing of the incremental. Any help is appreciated. Thanks, -- Dean Lambourn Senior Systems Administrator Pacific Coast Feather Company de...@pcf.com (206) 336-2332 -- October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register http://pubads.g.doubleclick.net/gampad/clk?id=60135991iu=/4140/ostg.clktrk ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register http://pubads.g.doubleclick.net/gampad/clk?id=60135991iu=/4140/ostg.clktrk ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- Dean Lambourn Senior Systems Administrator Pacific Coast Feather Company de...@pcf.com (206) 336-2332 -- October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register
Re: [BackupPC-users] Backuppc 3.3.0 - Full backups not backing up everything
On Mon, Oct 28, 2013 at 12:03 PM, Dean Lambourn de...@pcf.com wrote: Thanks you for your reply. The problem is with windows clients and we were backing these clients up with 3.1.0 with out experiencing this problem. It is more than one or two files not being backed up, it is that backuppc does not appear to even attempt backing up major directories such as Users so I doubt locked files are the problem. Are they windows junction points? Could be this bug: -- October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register http://pubads.g.doubleclick.net/gampad/clk?id=60135991iu=/4140/ostg.clktrk ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Backuppc 3.3.0 - Full backups not backing up everything
On Mon, Oct 28, 2013 at 12:27 PM, Les Mikesell lesmikes...@gmail.com wrote: On Mon, Oct 28, 2013 at 12:03 PM, Dean Lambourn de...@pcf.com wrote: Thanks you for your reply. The problem is with windows clients and we were backing these clients up with 3.1.0 with out experiencing this problem. It is more than one or two files not being backed up, it is that backuppc does not appear to even attempt backing up major directories such as Users so I doubt locked files are the problem. Are they windows junction points? Could be this bug: (sorry, Mac trackpad did something weird when I tried to past the link) https://bugzilla.redhat.com/show_bug.cgi?id=753531 -- Les Mikesell lesmikes...@gmail.com -- October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register http://pubads.g.doubleclick.net/gampad/clk?id=60135991iu=/4140/ostg.clktrk ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Utilizing posix_fadvise to not cache file contents
Hi, has there been any thoughts on utilizing posix_fadvise* in BackupPC? But here comes /proc/sys/vm/vfs_cache_pressure to the rescue Please tell us what results you get - if this approach works then that would avoid the need for application-level hints. Server and disk array specs: http://sourceforge.net/mailarchive/message.php?msg_id=31399081 /dev/sda is a RAID6 over 24 SAS disks a 600GB + hot spare, RAID config details as follows: Accelerator Ratio: 10% Read / 90% Write Drive Write Cache: Disabled Total Cache Size: 1024 MB Total Cache Memory Available: 912 MB No-Battery Write Cache: Disabled Cache Backup Power Source: Capacitors Logical Drive: 1 Size: 12.0 TB Fault Tolerance: RAID 6 (ADG) Heads: 255 Sectors Per Track: 32 Cylinders: 65535 Strip Size: 128 KB Full Stripe Size: 2816 KB Status: OK MultiDomain Status: OK Array Accelerator: Enabled Parity Initialization Status: Initialization Completed Disk Name: /dev/sda Mount Points: /var/lib/backuppc 12.0 TB Machine is backing up appr. 100 servers and now running for 7 days in production. version 3.2.1 Pool is 1037.96GB comprising 6278735 files and 4369 directories (as of 10/28 14:50), Pool hashing gives 963 repeated files with longest chain 13, Nightly cleanup removed 10328 files of size 4.53GB (around 10/28 14:50), Pool file system was recently at 9% (10/28 15:00), today's max is 9% (10/28 14:49) and yesterday's max was 10%. % df -i /var/lib/backuppc/ Filesystem InodesIUsed IFree IUse% Mounted on /dev/sda 2578525632 15105172 25634204601% /var/lib/backuppc % df -hT /var/lib/backuppc/ Filesystem Type Size Used Avail Use% Mounted on /dev/sda xfs13T 1.1T 11T 9% /var/lib/backuppc % xfs_info /dev/sda meta-data=/dev/sda isize=256agcount=13, agsize=268435424 blks = sectsz=512 attr=2 data = bsize=4096 blocks=3223157110, imaxpct=5 = sunit=32 swidth=704 blks naming =version 2 bsize=4096 ascii-ci=0 log =internal bsize=4096 blocks=521728, version=2 = sectsz=512 sunit=32 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 % grep backuppc /etc/fstab LABEL=Backuppc /var/lib/backuppc xfs noatime,nodiratime,nobarrier,inode64,logbufs=8,logbsize=256k 0 2 % cat /proc/sys/vm/vfs_cache_pressure 10 == default value=100, below 100 means: prefer dentries and inodes over page cache % echo 3 /proc/sys/vm/drop_caches == flushing dentries, inodes and page cache % time find /var/lib/backuppc/ -mtime -1 /dev/null real48m57.667s user0m50.619s sys 5m44.654s % slabtop -s c -o | head -n 30 Active / Total Objects (% used): 58157331 / 58185521 (100.0%) Active / Total Slabs (% used) : 5668137 / 5668137 (100.0%) Active / Total Caches (% used) : 88 / 178 (49.4%) Active / Total Size (% used) : 21272550.20K / 21278749.55K (100.0%) Minimum / Average / Maximum Object : 0.02K / 0.37K / 4096.00K OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 15105176 15105172 99%0.94K 37762944 15105176K xfs_inode 30130360 30130338 99%0.19K 1506518 20 6026072K dentry 901 9022178 99%0.06K 152919 59611676K size-64 1142640 1142599 99%0.31K 95220 12380880K xfs_buf 1886640 1886612 99%0.12K 62888 30251552K size-128 322574 322573 99%0.55K 460827184328K radix_tree_node 2063 2063 100% 16.00K20631 33008K size-16384 303400 302902 99%0.10K8200 37 32800K buffer_head 125840 125773 99%0.19K6292 20 25168K size-192 4818638656 80%0.21K2677 18 10708K xfs_ili 4704 4472 95%2.00K23522 9408K size-2048 6145 5221 84%0.73K12295 4916K ext2_inode_cache 2265322366 98%0.14K 839 27 3356K sysfs_dir_cache 5360 5336 99%0.50K 6708 2680K size-512 2268 2211 97%1.00K 5674 2268K size-1024 3661 3613 98%0.54K 5237 2092K inode_cache % free -m total used free sharedbuffers cached Mem: 96872 32741 64131 0 0 3733 -/+ buffers/cache: 29007 67865 Swap:0 0 0 == The 3733 MB cached are mostly pages from files accessed by backuppc during the 49 minutes 'find` ran (some backups were running). The 15.1 GB xfs_inode and 6 GB dentry cache are accumulated in the 32 GB used. Since now almost all inodes are cached in RAM,
Re: [BackupPC-users] OT: rsyncd as a service on Windows 8
I also have this issue on Windows 8 (and yet also run it flawlessly on Server 2012 going on a year now). I haven't fixed it yet, but in my case, I have noticed that the problem primarily exists on laptops. So my current running theory is that when the laptop goes to sleep, or hibernates, it screws with the service, and fails to restart, because the .pid file already exists. I've considered rewriting (whenever I have the time) the startup script to check for, and delete the .pid file before trying to start the actual binary. Thanks, --Mark -Original Message- From: Dan Johansson [mailto:dan.johans...@dmj.nu] Sent: Sunday, October 27, 2013 2:26 PM To: backuppc-users@lists.sourceforge.net Subject: Re: [BackupPC-users] OT: rsyncd as a service on Windows 8 On 26.10.2013 23:32, Timothy J Massey wrote: Dan Johansson dan.johans...@dmj.nu wrote on 10/26/2013 08:37:48 AM: Any suggestions on a) how to find out why rsyncd dies in the first place Not really: I have never run rsync on Windows 8. I *have* done it on Windows Server 2012 (based on Win8) with zero crashes across 3-4 servers on that OS and maybe 3 months of time. So it may not be a Windows 8 problem exactly, it may be just *your* Windows 8. Have you checked the Event Viewer? It usually shows you what's going on with rsync... b) how to fix this If we don't know why it dies how can we fix it? c) if this is unfixable how can I make rsyncd restart even if there are a .pid and .lock file around You can't, to my knowledge. I have wrapped launching rsync in a CMD to delete stale files before launching the daemon. (I hope I'm wrong: it would be nice not to need it!) Tim Massey Tim, do you mind sharing the CMD-file? Regards, -- Dan Johansson, http://www.dmj.nu *** This message is printed on 100% recycled electrons! *** -- October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register http://pubads.g.doubleclick.net/gampad/clk?id=60135991iu=/4140/ostg.clktrk ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] rsyncd full backup
Hi, I have implemented BackupPC for my customer. Initially, the backup transport is SMB, but recently, I noticed a lot of machine backup (full and incremental) is not able to complete in 8 hours, due to large number of file, and big file size. Last week, I installed DeltaCopy (rsycnd server for Windows) on one machine, and change the backup transport to rysncd. The backup runs well. But today, I noticed, when BackupPC is running a full backup on the machine that have rsyncd, it still takes 8 hours to do full backup. Which is I considered weird, because rsync suppose to compare that full backup, with the previous full backup (or previous full + incremental), so that only modified file is transferred. That is my expectation when I plan to use rsyncd. Any explanation why BackupPC is not running in this way? Any configuration that I can changed to make it work like what I expect? Thanks. -- Sharuzzaman Ahmat Raslan -- Android is increasing in popularity, but the open development platform that developers love is also attractive to malware creators. Download this white paper to learn more about secure code signing practices that can help keep Android apps secure. http://pubads.g.doubleclick.net/gampad/clk?id=65839951iu=/4140/ostg.clktrk___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] rsyncd full backup
On Mon, Oct 28, 2013 at 10:08 PM, Sharuzzaman Ahmat Raslan sharuzza...@gmail.com wrote: Hi, I have implemented BackupPC for my customer. Initially, the backup transport is SMB, but recently, I noticed a lot of machine backup (full and incremental) is not able to complete in 8 hours, due to large number of file, and big file size. Last week, I installed DeltaCopy (rsycnd server for Windows) on one machine, and change the backup transport to rysncd. The backup runs well. But today, I noticed, when BackupPC is running a full backup on the machine that have rsyncd, it still takes 8 hours to do full backup. Which is I considered weird, because rsync suppose to compare that full backup, with the previous full backup (or previous full + incremental), so that only modified file is transferred. That is my expectation when I plan to use rsyncd. Any explanation why BackupPC is not running in this way? Any configuration that I can changed to make it work like what I expect? Rsync will only transfer the changed data, but in full runs the contents of the files are read at both ends and compared with block checksums, so it takes some time. Incrementals runs will quickly skip files where the file timestamps and lengths are identical. See the section on 'Rsync checksum caching' in http://backuppc.sourceforge.net/faq/BackupPC.html regarding a way to avoid having to read/uncompress on the server side after 2 fulls have completed, but the data is always read on the target side. -- Les Mikesell lesmikes...@gmail.com -- Android is increasing in popularity, but the open development platform that developers love is also attractive to malware creators. Download this white paper to learn more about secure code signing practices that can help keep Android apps secure. http://pubads.g.doubleclick.net/gampad/clk?id=65839951iu=/4140/ostg.clktrk ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] rsyncd full backup
On 29/10/13 15:14, Sharuzzaman Ahmat Raslan wrote: Hi Les, Thanks.| | On Tue, Oct 29, 2013 at 11:33 AM, Les Mikesell lesmikes...@gmail.com mailto:lesmikes...@gmail.com wrote: On Mon, Oct 28, 2013 at 10:08 PM, Sharuzzaman Ahmat Raslan sharuzza...@gmail.com mailto:sharuzza...@gmail.com wrote: Hi, I have implemented BackupPC for my customer. Initially, the backup transport is SMB, but recently, I noticed a lot of machine backup (full and incremental) is not able to complete in 8 hours, due to large number of file, and big file size. Last week, I installed DeltaCopy (rsycnd server for Windows) on one machine, and change the backup transport to rysncd. The backup runs well. But today, I noticed, when BackupPC is running a full backup on the machine that have rsyncd, it still takes 8 hours to do full backup. Which is I considered weird, because rsync suppose to compare that full backup, with the previous full backup (or previous full + incremental), so that only modified file is transferred. That is my expectation when I plan to use rsyncd. Any explanation why BackupPC is not running in this way? Any configuration that I can changed to make it work like what I expect? Rsync will only transfer the changed data, but in full runs the contents of the files are read at both ends and compared with block checksums, so it takes some time. Incrementals runs will quickly skip files where the file timestamps and lengths are identical. See the section on 'Rsync checksum caching' in http://backuppc.sourceforge.net/faq/BackupPC.html regarding a way to avoid having to read/uncompress on the server side after 2 fulls have completed, but the data is always read on the target side. In essence, if I enable |--checksum-seed=32761 | then the rsync full backup will be faster? Yes, the third full backup after you enable that option will be faster *IF* the slow speed is due to the backup server needing to decompress the file and check the content. In the case that your backup client has really slow disk, then there is nothing you can do, except maybe modify backuppc for full backups to not send the ignore-times option to rsync (ie, every backup is an incremental). Or, of course, upgrade the client to improve performance. Regards, Adam -- Adam Goryachev Website Managers www.websitemanagers.com.au -- Android is increasing in popularity, but the open development platform that developers love is also attractive to malware creators. Download this white paper to learn more about secure code signing practices that can help keep Android apps secure. http://pubads.g.doubleclick.net/gampad/clk?id=65839951iu=/4140/ostg.clktrk___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
