Re: [BackupPC-users] Windows/Linux speed differences (windows is faster?)
Cameron writes:
> On 8/14/06, Craig Barratt <[EMAIL PROTECTED]> wrote:
> > Cameron writes:
> >
> > > I'm thinking of changing perms, owner, group, and maybe times all to
> > > no-OPTIONs. However, I'm concerned about how this will affect the
> > > program as I don't understand how File::RsyncP works, as it says in
> > > the comments. Can I go ahead and do this? Do I need times off or just
> > > the owner/group/perms off?
> >
> > File::RsyncP doesn't understand the -no options.
> > Just try removing:
> >
> > '--perms',
> > '--owner',
> > '--group',
> >
> > I have't tested that before, so please tell me whether
> > it works.
>
> I commented out those options, and did a full backup with Windows (so
> I guess commenting the options works fine). I got entries like this
> (lots of same and pool, a few create for empty file, and all the
> usuall create d for directories):
> create 644 0/0 0 Filename
> same 644 0/02249 Filename
> pool 644 0/0 1117184 Filename
> create d 755 0/0 0 Dirname
>
> So it looks like I dropped the user/group, but the file permissions
> are still there. Then I did an incremental in Windows, which was the
> same as before (short and sweet). Then a long incremental in Linux
> gave entries like this (lots of same, lots of create for empty file,
> the usual create d):
> create 555 0/0 0 Filename
> same 555 0/0 41812 Filename
> create d 555 0/0 0 Directory
>
> This is exactly as before, so removing the options from rsync really
> had no effect on the linux backup. Just for completeness, here are the
> backup size summaries, full (17) and first incr (18) are on Windows,
> other (19) is on linux:
> 17 full47312140.6 10.10 47312140.6
> 292 0.1
18 incr0 0.0 0.000 0.0 3 0.0
> 19incr47312140.6 16.59 47312140.6 5 0.0
>
> I feel like I'm closer, but I don't understand why rsync seems to be
> preserving the permissions on Windows even when I've told it not to.
> Any ideas? Do I need to empty my pool in order for the rsync option
> changes to be effective? Could this be related to how I'm mounting the
> Windows filesystem under linux (with options
> ro,users,auto,group,umask=0222)?
Yes, the umask will ensure that the file mode is different
in linux vs windows.
In any case, File::RsyncP ignores the --perms flag.
There are two places in the installed File/RsyncP.pm file (look
in perl's path from perl -V, or otherwise modify the file in a
File::RsyncP distribution and reinstall) where you will see this
line:
&& $f->{mode} == $attr->{mode}
You should replace both lines with something like this:
&& (!$rs->{rsyncOpts}{perms} || $f->{mode} == $attr->{mode})
I'll fix this in the next release of File::RsyncP.
Craig
-
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/
Re: [BackupPC-users] Windows/Linux speed differences (windows is faster?)
On 8/14/06, Craig Barratt <[EMAIL PROTECTED]> wrote: > Cameron writes: > > > I'm thinking of changing perms, owner, group, and maybe times all to > > no-OPTIONs. However, I'm concerned about how this will affect the > > program as I don't understand how File::RsyncP works, as it says in > > the comments. Can I go ahead and do this? Do I need times off or just > > the owner/group/perms off? > > File::RsyncP doesn't understand the -no options. > Just try removing: > > '--perms', > '--owner', > '--group', > > I have't tested that before, so please tell me whether > it works. I commented out those options, and did a full backup with Windows (so I guess commenting the options works fine). I got entries like this (lots of same and pool, a few create for empty file, and all the usuall create d for directories): create 644 0/0 0 Filename same 644 0/02249 Filename pool 644 0/0 1117184 Filename create d 755 0/0 0 Dirname So it looks like I dropped the user/group, but the file permissions are still there. Then I did an incremental in Windows, which was the same as before (short and sweet). Then a long incremental in Linux gave entries like this (lots of same, lots of create for empty file, the usual create d): create 555 0/0 0 Filename same 555 0/0 41812 Filename create d 555 0/0 0 Directory This is exactly as before, so removing the options from rsync really had no effect on the linux backup. Just for completeness, here are the backup size summaries, full (17) and first incr (18) are on Windows, other (19) is on linux: 17 full47312140.6 10.10 47312140.6 292 0.1 18 incr0 0.0 0.000 0.0 3 0.0 19 incr47312140.6 16.59 47312140.6 5 0.0 I feel like I'm closer, but I don't understand why rsync seems to be preserving the permissions on Windows even when I've told it not to. Any ideas? Do I need to empty my pool in order for the rsync option changes to be effective? Could this be related to how I'm mounting the Windows filesystem under linux (with options ro,users,auto,group,umask=0222)? Thanks again for all your help, Cameron - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] Windows/Linux speed differences (windows is faster?)
Cameron writes: > I'm thinking of changing perms, owner, group, and maybe times all to > no-OPTIONs. However, I'm concerned about how this will affect the > program as I don't understand how File::RsyncP works, as it says in > the comments. Can I go ahead and do this? Do I need times off or just > the owner/group/perms off? File::RsyncP doesn't understand the -no options. Just try removing: '--perms', '--owner', '--group', I have't tested that before, so please tell me whether it works. Craig - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] Windows/Linux speed differences (windows is faster?)
On 8/11/06, Craig Barratt <[EMAIL PROTECTED]> wrote:
> What doesn't make sense is that the two fulls have a lot less files
> than the incremental. I suspect you setup a small test for backups
> #0 and #1, then set it to backup a lot more prior to incremental #2.
> Therefore each incremental is backing up a lot of files not in the
> full. You should start a full backup and then see what happens
> with the next incrementals.
Ahh, now it all makes sense. I had considered this, but I was under
the impression that incremental meant files changed since the last
backup (incremental or full). Now I see that it means files changed
since the last full only. Manually scheduling a full backup caused all
future incrementals to be smaller. Thanks for setting me straight.
> My original claim still stands: on the dual boot system I
> suspect the uid/gid or mtime is not returned consistently
> when your machine is booted on windows vs linux. Therefore,
> if the last full was from windows, then a linux incremental
> will backup every file again (and vica versa). With rsync
> not a lot of data will be transferred, but it will take a
> lot more time. I suggested you manually run rsync in each
> case to see.
It looks like this is the case. Looking at the Xferlog files, the
initial full backup on Windows has entries like this:
create 64418/544 13824 Filename
Whereas the incrementals in linux had entries like this for the same file:
same 555 0/0 13824 Filename
Is there any way to make rsync disregard the permissions just for the
dual-boot backup? I have the following in my config.pl file:
#
# Arguments to rsync for backup. Do not edit the first set unless you
# have a thorough understanding of how File::RsyncP works.
#
# Examples of additional arguments that should work are --exclude/--include,
# eg:
#
# $Conf{RsyncArgs} = [
# # original arguments here
# '-v',
# '--exclude', '/proc',
# '--exclude', '*.tmp',
# ];
#
$Conf{RsyncArgs} = [
#
# Do not edit these!
#
'--numeric-ids',
'--perms',
'--owner',
'--group',
'--devices',
'--links',
'--times',
'--block-size=2048',
'--recursive',
#
# If you are using a patched client rsync that supports the
# --checksum-seed option (see http://backuppc.sourceforge.net),
# then uncomment this to enabled rsync checksum cachcing
#
'--checksum-seed=32761',
#
# Add additional arguments here
#
'-D',
];
I'm thinking of changing perms, owner, group, and maybe times all to
no-OPTIONs. However, I'm concerned about how this will affect the
program as I don't understand how File::RsyncP works, as it says in
the comments. Can I go ahead and do this? Do I need times off or just
the owner/group/perms off?
Thanks again for all your help,
Cameron
-
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/
Re: [BackupPC-users] Windows/Linux speed differences (windows is faster?)
Cameron Dale writes: > Craig Barratt said the following on 11/08/2006 1:47 AM: > > Yes, the entire set of files is being transferred on an > > incremental with a linux boot. > > But why is this happening? What is the difference between Windows and > Linux that would cause this? > > > That means some meta data (eg: uid, gid, mtime, size) is being > > delivered differently to rsync on windows vs linux. > > > > It is quite possible that the uid and gid are different when you boot > > windows vs linux. > > As I said in my post, this problem is easiest to see on the dual-boot > machine, however it also is apparent on other machines. > > Here is the backup listing for a Windows machine, where you can see that > the number of files transferred after the first backup is quite small: > >Totals Existing Files New Files > Backup# Type #Files Size/MB MB/sec #Files Size/MB #Files Size/MB > 0 full800 676.7 2.2387 47.4795 629.3 > 1 incr5 34.90.942 9.2 21 25.7 > 2 incr5 34.91.202 3.0 8 31.9 > 3 incr5 34.90.856 34.90 0.0 > 4 incr11 35.40.964 19.317 16.0 > 5 incr11 35.40.4712 35.40 0.0 > 6 incr11 35.40.9112 35.40 0.0 > 7 incr11 35.41.4112 35.40 0.0 Yes, this case looks normal. > Here is the backup listing for a Linux machine, where the incrementals > are getting bigger and bigger each time: > >Totals Existing Files New Files > Backup# Type #Files Size/MB MB/sec #Files Size/MB #Files Size/MB > 0 full23505.3 0.35397 0.1 22355.3 > 1 full26335.7 0.3823515.2 576 0.5 > 2 incr14982 187.1 0.8210703 36.15832151.5 > 3 incr14982 187.6 1.2614944 159.7 47 27.9 > 4 incr14984 187.9 1.2414943 161.7 53 26.3 > 5 incr15705 194.0 0.9615650 166.6 370 27.6 > 6 incr15705 194.9 1.0515666 166.3 48 28.6 > 7 incr15705 195.4 1.0715670 167.2 43 28.3 > 8 incr16207 197.4 1.3716060 167.0 268 30.4 > 9 incr16211 197.8 1.5816172 168.9 57 29.0 > 10incr16240 201.8 1.5416199 169.9 51 31.8 > 11incr16240 202.2 1.5816202 172.0 48 30.2 > 12incr16241 188.5 1.4716200 168.6 51 19.9 > 13incr16245 189.5 1.3916201 170.7 56 18.9 > 14incr16251 190.6 1.2716203 170.7 64 19.9 > 15incr16253 191.7 1.1716213 171.0 53 20.7 What doesn't make sense is that the two fulls have a lot less files than the incremental. I suspect you setup a small test for backups #0 and #1, then set it to backup a lot more prior to incremental #2. Therefore each incremental is backing up a lot of files not in the full. You should start a full backup and then see what happens with the next incrementals. > > It might be related to file time stamps. What type > > of file system is this? If it is FAT then you are likely a victim of > > the DST problem. Google "rsync FAT DST". > > Nope, it's NTFS on Windows, ReiserFS on Linux. My original claim still stands: on the dual boot system I suspect the uid/gid or mtime is not returned consistently when your machine is booted on windows vs linux. Therefore, if the last full was from windows, then a linux incremental will backup every file again (and vica versa). With rsync not a lot of data will be transferred, but it will take a lot more time. I suggested you manually run rsync in each case to see. Bottom line: you are seeing two different issues here. Craig - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] Windows/Linux speed differences (windows is faster?)
Craig Barratt said the following on 11/08/2006 1:47 AM: > Yes, the entire set of files is being transferred on an > incremental with a linux boot. But why is this happening? What is the difference between Windows and Linux that would cause this? > That means some meta data (eg: uid, gid, mtime, size) is being > delivered differently to rsync on windows vs linux. > > It is quite possible that the uid and gid are different when you boot > windows vs linux. As I said in my post, this problem is easiest to see on the dual-boot machine, however it also is apparent on other machines. Here is the backup listing for a Windows machine, where you can see that the number of files transferred after the first backup is quite small: Totals Existing Files New Files Backup# Type#Files Size/MB MB/sec #Files Size/MB #Files Size/MB 0 full800 676.7 2.2387 47.4795 629.3 1 incr5 34.90.942 9.2 21 25.7 2 incr5 34.91.202 3.0 8 31.9 3 incr5 34.90.856 34.90 0.0 4 incr11 35.40.964 19.317 16.0 5 incr11 35.40.4712 35.40 0.0 6 incr11 35.40.9112 35.40 0.0 7 incr11 35.41.4112 35.40 0.0 Here is the backup listing for a Linux machine, where the incrementals are getting bigger and bigger each time: Totals Existing Files New Files Backup# Type#Files Size/MB MB/sec #Files Size/MB #Files Size/MB 0 full23505.3 0.35397 0.1 22355.3 1 full26335.7 0.3823515.2 576 0.5 2 incr14982 187.1 0.8210703 36.15832151.5 3 incr14982 187.6 1.2614944 159.7 47 27.9 4 incr14984 187.9 1.2414943 161.7 53 26.3 5 incr15705 194.0 0.9615650 166.6 370 27.6 6 incr15705 194.9 1.0515666 166.3 48 28.6 7 incr15705 195.4 1.0715670 167.2 43 28.3 8 incr16207 197.4 1.3716060 167.0 268 30.4 9 incr16211 197.8 1.5816172 168.9 57 29.0 10 incr16240 201.8 1.5416199 169.9 51 31.8 11 incr16240 202.2 1.5816202 172.0 48 30.2 12 incr16241 188.5 1.4716200 168.6 51 19.9 13 incr16245 189.5 1.3916201 170.7 56 18.9 14 incr16251 190.6 1.2716203 170.7 64 19.9 15 incr16253 191.7 1.1716213 171.0 53 20.7 > It might be related to file time stamps. What type > of file system is this? If it is FAT then you are likely a victim of > the DST problem. Google "rsync FAT DST". Nope, it's NTFS on Windows, ReiserFS on Linux. > I recommend doing some manual rsyncs and seeing what meta data > you get after the transer with windows vs linux. > > Craig > -- Cameron Dale [EMAIL PROTECTED] - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] Windows/Linux speed differences (windows is faster?)
Cameron Dale writes: > I'm backing up several different machines on my local network to my > debian-based server using BackupPC. I'm using rsyncd for all of this, > 2.6.8 on the Linux machines, and cygwin-rsyncd-2.6.2_0.zip on the > Windows machines. One of the machines is even dual booted, and the > same data is backed up sometimes by Windows, sometimes by Linux. > > I'm seeing differences in the speed and number of files transferred > for incremental backups on Linux compared to Windows. Looking at the > XferLog files, it seems that the Windows machines are only > transferring the changed files (the log contains "create d" for all > the directories, and "create" for a few new files), whereas the Linux > machines transfer all the files (the log contains similar "create d" > and "create" entries, as well as "same" entries for ALL the unchanged > files). > > This is most apparent on the dual-boot machine as some incrementals > are small and fast, whereas others are large and slow. Here is the > backup listing for that machine. Note that backups 1, 3, 7, and 10 > were all when it was booted Linux, the others were all Windows. > > Totals Existing Files New Files > Backup# Type #Files Size/MB MB/sec #Files Size/MB #Files Size/MB > 0 full49052239.8 3.21225 8.2 50722231.7 > 1 incr49052239.8 10.37 49062239.8 391 0.1 > 2 incr7 5.7 0.0818 1.8 42 3.9 > 3 incr49052239.5 3.9449042239.2 3 0.4 > 4 incr8 5.8 0.082 2.0 10 3.8 > 5 incr8 5.9 0.082 2.0 8 3.8 > 6 incr8 5.9 0.092 2.0 10 3.9 > 7 incr49052239.7 12.65 49002235.9 9 3.8 > 8 incr8 5.9 0.097 5.6 2 0.4 > 9 incr34 24.60.3526 20.214 4.4 > 10incr47302140.5 18.61 47252136.6 11 3.9 > 11incr34 24.70.3728 20.611 4.0 > > I suspect this is somehow related to checksum caching, but I have > enabled the checksum-seed option globally and I think all the rsync's > I'm using should have it available. Is there something else in the > cygwin-rsyncd-2.6.2_0.zip version of rsync that is speeding up the > Windows backups? Can I somehow get that on my Linux machines too? Yes, the entire set of files is being transferred on an incremental with a linux boot. That means some meta data (eg: uid, gid, mtime, size) is being delivered differently to rsync on windows vs linux. It is quite possible that the uid and gid are different when you boot windows vs linux. It might be related to file time stamps. What type of file system is this? If it is FAT then you are likely a victim of the DST problem. Google "rsync FAT DST". I recommend doing some manual rsyncs and seeing what meta data you get after the transer with windows vs linux. Craig - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
[BackupPC-users] Windows/Linux speed differences (windows is faster?)
I'm backing up several different machines on my local network to my debian-based server using BackupPC. I'm using rsyncd for all of this, 2.6.8 on the Linux machines, and cygwin-rsyncd-2.6.2_0.zip on the Windows machines. One of the machines is even dual booted, and the same data is backed up sometimes by Windows, sometimes by Linux. I'm seeing differences in the speed and number of files transferred for incremental backups on Linux compared to Windows. Looking at the XferLog files, it seems that the Windows machines are only transferring the changed files (the log contains "create d" for all the directories, and "create" for a few new files), whereas the Linux machines transfer all the files (the log contains similar "create d" and "create" entries, as well as "same" entries for ALL the unchanged files). This is most apparent on the dual-boot machine as some incrementals are small and fast, whereas others are large and slow. Here is the backup listing for that machine. Note that backups 1, 3, 7, and 10 were all when it was booted Linux, the others were all Windows. Totals Existing Files New Files Backup# Type#Files Size/MB MB/sec #Files Size/MB #Files Size/MB 0 full49052239.8 3.21225 8.2 50722231.7 1 incr49052239.8 10.37 49062239.8 391 0.1 2 incr7 5.7 0.0818 1.8 42 3.9 3 incr49052239.5 3.9449042239.2 3 0.4 4 incr8 5.8 0.082 2.0 10 3.8 5 incr8 5.9 0.082 2.0 8 3.8 6 incr8 5.9 0.092 2.0 10 3.9 7 incr49052239.7 12.65 49002235.9 9 3.8 8 incr8 5.9 0.097 5.6 2 0.4 9 incr34 24.60.3526 20.214 4.4 10 incr47302140.5 18.61 47252136.6 11 3.9 11 incr34 24.70.3728 20.611 4.0 I suspect this is somehow related to checksum caching, but I have enabled the checksum-seed option globally and I think all the rsync's I'm using should have it available. Is there something else in the cygwin-rsyncd-2.6.2_0.zip version of rsync that is speeding up the Windows backups? Can I somehow get that on my Linux machines too? Thanks for the help, Cam - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
