Re: [BackupPC-users] backuppc incremental taking up a lot of bandwidth with no additional changes
On 1/20/2012 4:49 PM, Tim Fletcher wrote: > On Fri, 2012-01-20 at 16:11 -0800, smallpox wrote: >> On 1/20/2012 11:49 AM, Jeffrey J. Kosowsky wrote: >>> Second, what makes you think the issue is a bandwidth issue? You just >>> said that it takes 10 minutes. Have you determined that the time is >>> due to bandwidth bottlenecks and not just disk reads and rsync >>> computations? >> I see mrtg's graph, it's doing about 5mbit for 10 minutes. > Could the problem be caused by the granularity of timestamps[1] on > Windows filesystems? > > Try adding --modify-window=1 to the rsync command line didn't help, I stopped it. it was killing my ability to work on other things. > > [1]ftp://pserver.samba.org/pub/unpacked/rsyncweb/daylight-savings.html > -- Try before you buy = See our experts in action! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-dev2 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] backuppc incremental taking up a lot of bandwidth with no additional changes
On 1/20/2012 4:49 PM, Tim Fletcher wrote: > On Fri, 2012-01-20 at 16:11 -0800, smallpox wrote: >> On 1/20/2012 11:49 AM, Jeffrey J. Kosowsky wrote: >>> Second, what makes you think the issue is a bandwidth issue? You just >>> said that it takes 10 minutes. Have you determined that the time is >>> due to bandwidth bottlenecks and not just disk reads and rsync >>> computations? >> I see mrtg's graph, it's doing about 5mbit for 10 minutes. > Could the problem be caused by the granularity of timestamps[1] on > Windows filesystems? > > Try adding --modify-window=1 to the rsync command line > > [1]ftp://pserver.samba.org/pub/unpacked/rsyncweb/daylight-savings.html Would I need to revert to level 1 for incremental ? it only seems right to, to test it. -- Try before you buy = See our experts in action! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-dev2 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] backuppc incremental taking up a lot of bandwidth with no additional changes
On Fri, 2012-01-20 at 16:11 -0800, smallpox wrote: > > On 1/20/2012 11:49 AM, Jeffrey J. Kosowsky wrote: > > Second, what makes you think the issue is a bandwidth issue? You just > > said that it takes 10 minutes. Have you determined that the time is > > due to bandwidth bottlenecks and not just disk reads and rsync > > computations? > > I see mrtg's graph, it's doing about 5mbit for 10 minutes. Could the problem be caused by the granularity of timestamps[1] on Windows filesystems? Try adding --modify-window=1 to the rsync command line [1]ftp://pserver.samba.org/pub/unpacked/rsyncweb/daylight-savings.html -- Tim Fletcher -- Try before you buy = See our experts in action! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-dev2 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] backuppc incremental taking up a lot of bandwidth with no additional changes
51 incr no 1 1/20 06:45 10.4 0.4 /home/backuppc/BackupPC-3.2.1/pc/danger/51 52 incr no 2 1/20 07:00 0.2 0.4 /home/backuppc/BackupPC-3.2.1/pc/danger/52 53 incr no 2 1/20 07:15 0.2 0.4 /home/backuppc/BackupPC-3.2.1/pc/danger/53 54 incr no 3 1/20 07:30 0.2 0.4 /home/backuppc/BackupPC-3.2.1/pc/danger/54 55 incr no 4 1/20 07:45 0.2 0.4 /home/backuppc/BackupPC-3.2.1/pc/danger/55 56 incr no 5 1/20 08:00 0.2 0.3 /home/backuppc/BackupPC-3.2.1/pc/danger/56 57 incr no 6 1/20 08:15 0.2 0.3 /home/backuppc/BackupPC-3.2.1/pc/danger/57 58 incr no 2 1/20 08:30 0.2 0.3 /home/backuppc/BackupPC-3.2.1/pc/danger/58 59 incr no 3 1/20 08:45 0.2 0.3 /home/backuppc/BackupPC-3.2.1/pc/danger/59 60 incr no 4 1/20 09:00 0.2 0.3 /home/backuppc/BackupPC-3.2.1/pc/danger/60 61 incr no 5 1/20 09:15 0.2 0.3 /home/backuppc/BackupPC-3.2.1/pc/danger/61 62 incr no 6 1/20 09:30 0.2 0.3 /home/backuppc/BackupPC-3.2.1/pc/danger/62 63 incr no 2 1/20 09:45 0.2 0.3 /home/backuppc/BackupPC-3.2.1/pc/danger/63 64 incr no 3 1/20 10:00 0.2 0.3 /home/backuppc/BackupPC-3.2.1/pc/danger/64 65 incr no 4 1/20 10:15 0.2 0.2 /home/backuppc/BackupPC-3.2.1/pc/danger/65 66 incr no 5 1/20 10:30 0.2 0.2 /home/backuppc/BackupPC-3.2.1/pc/danger/66 67 incr no 6 1/20 10:45 0.2 0.2 /home/backuppc/BackupPC-3.2.1/pc/danger/67 68 incr no 2 1/20 11:00 0.2 0.2 /home/backuppc/BackupPC-3.2.1/pc/danger/68 69 incr no 3 1/20 11:15 0.2 0.2 /home/backuppc/BackupPC-3.2.1/pc/danger/69 70 incr no 4 1/20 11:30 0.2 0.2 /home/backuppc/BackupPC-3.2.1/pc/danger/70 71 incr no 5 1/20 11:45 0.2 0.2 /home/backuppc/BackupPC-3.2.1/pc/danger/71 72 incr no 6 1/20 12:00 0.2 0.2 /home/backuppc/BackupPC-3.2.1/pc/danger/72 73 incr no 2 1/20 12:15 0.2 0.2 /home/backuppc/BackupPC-3.2.1/pc/danger/73 74 incr no 3 1/20 12:30 0.2 0.2 /home/backuppc/BackupPC-3.2.1/pc/danger/74 75 incr no 4 1/20 12:45 0.2 0.1 /home/backuppc/BackupPC-3.2.1/pc/danger/75 76 incr no 5 1/20 13:00 0.2 0.1 /home/backuppc/BackupPC-3.2.1/pc/danger/76 77 incr no 6 1/20 13:15 0.2 0.1 /home/backuppc/BackupPC-3.2.1/pc/danger/77 78 incr no 2 1/20 13:30 0.2 0.1 /home/backuppc/BackupPC-3.2.1/pc/danger/78 79 incr no 3 1/20 13:45 0.2 0.1 /home/backuppc/BackupPC-3.2.1/pc/danger/79 80 incr no 4 1/20 14:00 0.2 0.1 /home/backuppc/BackupPC-3.2.1/pc/danger/80 81 incr no 5 1/20 14:15 0.2 0.1 /home/backuppc/BackupPC-3.2.1/pc/danger/81 82 incr no 6 1/20 14:30 0.2 0.1 /home/backuppc/BackupPC-3.2.1/pc/danger/82 83 incr no 2 1/20 14:45 0.2 0.1 /home/backuppc/BackupPC-3.2.1/pc/danger/83 84 incr no 3 1/20 15:00 0.2 0.0 /home/backuppc/BackupPC-3.2.1/pc/danger/84 85 incr no 4 1/20 15:15 0.2 0.0 /home/backuppc/BackupPC-3.2.1/pc/danger/85 86 incr no 5 1/20 15:30 0.2 0.0 /home/backuppc/BackupPC-3.2.1/pc/danger/86 87 incr no 6 1/20 15:45 0.2 0.0 /home/backuppc/BackupPC-3.2.1/pc/danger/87 88 incr no 2 1/20 16:00 0.2 0.0 /home/backuppc/BackupPC-3.2.1/pc/danger/88 so at 7, i changed the levels from 1,2,3,4,5,6 to 2,3,4,5,6 and this is what i get. i do not understand how this is happening though. thanks On 1/20/2012 10:02 AM, Les Mikesell wrote: > On Fri, Jan 20, 2012 at 11:16 AM, smallpox wrote: >> understood, there arent a lot of files currently, nothing has changed >> since day 1. >> >> its only 400 mb. >> >> if i have level 2, it does incremental in 10 seconds, why does that >> happen? level 1 takes 10 min > That doesn't make much sense to me for an rsync type backup. I would > expect the level2 to have to do more work server-side to merge the > directory info of the prior full and incremental. A level0 (full) > should take much longer to read all the data but not much more > bandwidth to exchange the block checksums. You might improve your > overall timing if you can split the runs into several top level > directories, using different host entries with a ClientNameAlias > pointing back to the same host. That would let you overlap runs and > skew the days when fulls happen.
Re: [BackupPC-users] backuppc incremental taking up a lot of bandwidth with no additional changes
On 1/20/2012 11:49 AM, Jeffrey J. Kosowsky wrote: > smallpox wrote at about 07:15:07 -0800 on Friday, January 20, 2012: > > i was under the impression that rsync does the comparison with little or > > no bandwidth. > > First, PLEASE DON'T TOP-POST - it makes following and responding > to a thread near impossible. Sorry about that. > Second, what makes you think the issue is a bandwidth issue? You just > said that it takes 10 minutes. Have you determined that the time is > due to bandwidth bottlenecks and not just disk reads and rsync > computations? I see mrtg's graph, it's doing about 5mbit for 10 minutes. > > Third, why would you expect rsync -- even without backuppc -- to take > significantly fewer than 10 minutes to crawl a directory tree of > "a hundred thousand files" including in reading in the inode > information from each file, transmitting it across the network, > comparing it to said information stored in the BackupPC attrib > file. Plus any changed files will require reading, decompressing, and > computing rolling md4 checksums on each of the target files as part of > the rsync algorithm. Then the full directory tree needs to be > constructed on the BackupPC Server including attrib files and any > changed files (this is for incrementals). Do you expect this to all > happen instantaneously? I do not have hundreds of thousands of files yet. 141 Files, 20 Folders, 388 MB (407,536,068 bytes) > > Fourth, are you sure that you want to do "continuous" (your words) > backups? That means that you are creating an ever-increasing (and > theoretically infinite) number of incremental backups each requiring a > full directory tree parallel to the source plus attrib files and > changed files. Additionally, any time a file changes by even one byte, > than a whole new copy is saved to the pool (think log files or system > files that may change multiple times per second). > > In summary, I think you are trying to solve a problem that may not > need to be solved, using a tool that is not meant to solve it, without > understanding what is causing your problems and without knowing how > the tool actually works in the first place :) I am trying to mimic Iron Mountain's software which runs as some software on the system and sends differences every 15 minutes. > > > > > On 1/20/2012 5:50 AM, Michael Stowe wrote: > > >> BackupPC 3.2.1 > > >> Windows 7 rsyncd over ssh, west coast > > >> the server is in the east. > > >> > > >> my goal is to have it updated every 15 minutes, i've gotten that to > work > > >> but for incremental, with no changes, it's still taking about 10 > minutes > > >> most of the time and it is doing traffic. > > > I would expect most of that traffic would be checking to see if there > are > > > changes to the files. > > > > > >> i do not understand why incremental with absolutely no changes is > taking > > >> so long ? ultimately i'd like to have a hundred thousand files being > > >> backed up constantly, is this just a dream ? > > > How would you propose that changes be recognized, if not through > rsync's > > > mechanisms of comparison? > > > > > > You can always go to cmd/cifs and use time stamps to determine what's > > > changed, I suppose. > > > > > >> thanks in advance > > > > > > > -- > > > Keep Your Developer Skills Current with LearnDevNow! > > > The most comprehensive online learning library for Microsoft > developers > > > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, > MVC3, > > > Metro Style Apps, more. Free future releases when you subscribe now! > > > http://p.sf.net/sfu/learndevnow-d2d > > > ___ > > > BackupPC-users mailing list > > > BackupPC-users@lists.sourceforge.net > > > List:https://lists.sourceforge.net/lists/listinfo/backuppc-users > > > Wiki:http://backuppc.wiki.sourceforge.net > > > Project: http://backuppc.sourceforge.net/ > > > > > > > > -- > > Keep Your Developer Skills Current with LearnDevNow! > > The most comprehensive online learning library for Microsoft developers > > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, > > Metro Style Apps, more. Free future releases when you subscribe now! > > http://p.sf.net/sfu/learndevnow-d2d > > ___ > > BackupPC-users mailing list > > BackupPC-users@lists.sourceforge.net > > List:https://lists.sourceforge.net/lists/listinfo/backuppc-users > > Wiki:http://backuppc.wiki.sourceforge.net > > Project: http://backuppc.sourceforge.net/ > > -- > Keep Your Developer Skills Cur
Re: [BackupPC-users] backuppc incremental taking up a lot of bandwidth with no additional changes
> i was under the impression that rsync does the comparison with little or > no bandwidth. Little bandwidth, yes (although this is relative, naturally) but taking little bandwidth and taking little *time* are two entirely different things. Rsync on each end is calculating and exchanging hashes. > On 1/20/2012 5:50 AM, Michael Stowe wrote: >>> BackupPC 3.2.1 >>> Windows 7 rsyncd over ssh, west coast >>> the server is in the east. >>> >>> my goal is to have it updated every 15 minutes, i've gotten that to >>> work >>> but for incremental, with no changes, it's still taking about 10 >>> minutes >>> most of the time and it is doing traffic. >> I would expect most of that traffic would be checking to see if there >> are >> changes to the files. >> >>> i do not understand why incremental with absolutely no changes is >>> taking >>> so long ? ultimately i'd like to have a hundred thousand files being >>> backed up constantly, is this just a dream ? >> How would you propose that changes be recognized, if not through rsync's >> mechanisms of comparison? >> >> You can always go to cmd/cifs and use time stamps to determine what's >> changed, I suppose. >> >>> thanks in advance >> >> -- >> Keep Your Developer Skills Current with LearnDevNow! >> The most comprehensive online learning library for Microsoft developers >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, >> Metro Style Apps, more. Free future releases when you subscribe now! >> http://p.sf.net/sfu/learndevnow-d2d >> ___ >> BackupPC-users mailing list >> BackupPC-users@lists.sourceforge.net >> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users >> Wiki:http://backuppc.wiki.sourceforge.net >> Project: http://backuppc.sourceforge.net/ >> > -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] backuppc incremental taking up a lot of bandwidth with no additional changes
On 01/20/2012 09:07 PM, Jeffrey J. Kosowsky wrote: > Stefan Peter wrote at about 21:00:22 +0100 on Friday, January 20, 2012: > > On 01/20/2012 08:49 PM, Jeffrey J. Kosowsky wrote: > > > In summary, I think you are trying to solve a problem that may not > > > need to be solved, using a tool that is not meant to solve it, without > > > understanding what is causing your problems and without knowing how > > > the tool actually works in the first place :) > > > > can I use this sentence for my own purposes or do you have a copyright > > on it? > > Sure use it to your hearts content... but if anyone asks you where it > came from, I just made it up on the spur of the moment... I will remind in your debt forever. Regards Stefan Peter -- In theory there is no difference between theory and practice. In practice there is. -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] backuppc incremental taking up a lot of bandwidth with no additional changes
On Fri, Jan 20, 2012 at 2:07 PM, Jeffrey J. Kosowsky wrote: > Stefan Peter wrote at about 21:00:22 +0100 on Friday, January 20, 2012: > > On 01/20/2012 08:49 PM, Jeffrey J. Kosowsky wrote: > > > In summary, I think you are trying to solve a problem that may not > > > need to be solved, using a tool that is not meant to solve it, without > > > understanding what is causing your problems and without knowing how > > > the tool actually works in the first place :) > > > > can I use this sentence for my own purposes or do you have a copyright > > on it? > > Sure use it to your hearts content... but if anyone asks you where it > came from, I just made it up on the spur of the moment... Seems true in general for everything invented in at least the last hundred years, though. -- Les Mikesell lesmikes...@gmail.com -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] backuppc incremental taking up a lot of bandwidth with no additional changes
On 20-Jan-12 12:07, Jeffrey J. Kosowsky wrote: > Stefan Peter wrote at about 21:00:22 +0100 on Friday, January 20, 2012: > > On 01/20/2012 08:49 PM, Jeffrey J. Kosowsky wrote: > > > In summary, I think you are trying to solve a problem that may not > > > need to be solved, using a tool that is not meant to solve it, without > > > understanding what is causing your problems and without knowing how > > > the tool actually works in the first place :) > > > > can I use this sentence for my own purposes or do you have a copyright > > on it? > > Sure use it to your hearts content... but if anyone asks you where it > came from, I just made it up on the spur of the moment... That's beautiful. And applicable to so many situations I deal with. -- Steve Willoughby / st...@alchemy.com "A ship in harbor is safe, but that is not what ships are built for." PGP Fingerprint 4615 3CCE 0F29 AE6C 8FF4 CA01 73FE 997A 765D 696C -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] backuppc incremental taking up a lot of bandwidth with no additional changes
Stefan Peter wrote at about 21:00:22 +0100 on Friday, January 20, 2012: > On 01/20/2012 08:49 PM, Jeffrey J. Kosowsky wrote: > > In summary, I think you are trying to solve a problem that may not > > need to be solved, using a tool that is not meant to solve it, without > > understanding what is causing your problems and without knowing how > > the tool actually works in the first place :) > > can I use this sentence for my own purposes or do you have a copyright > on it? Sure use it to your hearts content... but if anyone asks you where it came from, I just made it up on the spur of the moment... -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] backuppc incremental taking up a lot of bandwidth with no additional changes
On 01/20/2012 08:49 PM, Jeffrey J. Kosowsky wrote: > In summary, I think you are trying to solve a problem that may not > need to be solved, using a tool that is not meant to solve it, without > understanding what is causing your problems and without knowing how > the tool actually works in the first place :) can I use this sentence for my own purposes or do you have a copyright on it? Regards Stefan Peter -- In theory there is no difference between theory and practice. In practice there is. -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] backuppc incremental taking up a lot of bandwidth with no additional changes
smallpox wrote at about 07:15:07 -0800 on Friday, January 20, 2012: > i was under the impression that rsync does the comparison with little or > no bandwidth. First, PLEASE DON'T TOP-POST - it makes following and responding to a thread near impossible. Second, what makes you think the issue is a bandwidth issue? You just said that it takes 10 minutes. Have you determined that the time is due to bandwidth bottlenecks and not just disk reads and rsync computations? Third, why would you expect rsync -- even without backuppc -- to take significantly fewer than 10 minutes to crawl a directory tree of "a hundred thousand files" including in reading in the inode information from each file, transmitting it across the network, comparing it to said information stored in the BackupPC attrib file. Plus any changed files will require reading, decompressing, and computing rolling md4 checksums on each of the target files as part of the rsync algorithm. Then the full directory tree needs to be constructed on the BackupPC Server including attrib files and any changed files (this is for incrementals). Do you expect this to all happen instantaneously? Fourth, are you sure that you want to do "continuous" (your words) backups? That means that you are creating an ever-increasing (and theoretically infinite) number of incremental backups each requiring a full directory tree parallel to the source plus attrib files and changed files. Additionally, any time a file changes by even one byte, than a whole new copy is saved to the pool (think log files or system files that may change multiple times per second). In summary, I think you are trying to solve a problem that may not need to be solved, using a tool that is not meant to solve it, without understanding what is causing your problems and without knowing how the tool actually works in the first place :) > On 1/20/2012 5:50 AM, Michael Stowe wrote: > >> BackupPC 3.2.1 > >> Windows 7 rsyncd over ssh, west coast > >> the server is in the east. > >> > >> my goal is to have it updated every 15 minutes, i've gotten that to work > >> but for incremental, with no changes, it's still taking about 10 minutes > >> most of the time and it is doing traffic. > > I would expect most of that traffic would be checking to see if there are > > changes to the files. > > > >> i do not understand why incremental with absolutely no changes is taking > >> so long ? ultimately i'd like to have a hundred thousand files being > >> backed up constantly, is this just a dream ? > > How would you propose that changes be recognized, if not through rsync's > > mechanisms of comparison? > > > > You can always go to cmd/cifs and use time stamps to determine what's > > changed, I suppose. > > > >> thanks in advance > > > > -- > > Keep Your Developer Skills Current with LearnDevNow! > > The most comprehensive online learning library for Microsoft developers > > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, > > Metro Style Apps, more. Free future releases when you subscribe now! > > http://p.sf.net/sfu/learndevnow-d2d > > ___ > > BackupPC-users mailing list > > BackupPC-users@lists.sourceforge.net > > List:https://lists.sourceforge.net/lists/listinfo/backuppc-users > > Wiki:http://backuppc.wiki.sourceforge.net > > Project: http://backuppc.sourceforge.net/ > > > > -- > Keep Your Developer Skills Current with LearnDevNow! > The most comprehensive online learning library for Microsoft developers > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, > Metro Style Apps, more. Free future releases when you subscribe now! > http://p.sf.net/sfu/learndevnow-d2d > ___ > BackupPC-users mailing list > BackupPC-users@lists.sourceforge.net > List:https://lists.sourceforge.net/lists/listinfo/backuppc-users > Wiki:http://backuppc.wiki.sourceforge.net > Project: http://backuppc.sourceforge.net/ -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] backuppc incremental taking up a lot of bandwidth with no additional changes
On Fri, Jan 20, 2012 at 11:16 AM, smallpox wrote: > understood, there arent a lot of files currently, nothing has changed > since day 1. > > its only 400 mb. > > if i have level 2, it does incremental in 10 seconds, why does that > happen? level 1 takes 10 min That doesn't make much sense to me for an rsync type backup. I would expect the level2 to have to do more work server-side to merge the directory info of the prior full and incremental. A level0 (full) should take much longer to read all the data but not much more bandwidth to exchange the block checksums. You might improve your overall timing if you can split the runs into several top level directories, using different host entries with a ClientNameAlias pointing back to the same host. That would let you overlap runs and skew the days when fulls happen. -- Les Mikesell lesmikes...@gmail.com -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] backuppc incremental taking up a lot of bandwidth with no additional changes
understood, there arent a lot of files currently, nothing has changed since day 1. its only 400 mb. if i have level 2, it does incremental in 10 seconds, why does that happen? level 1 takes 10 min On 1/20/2012 8:19 AM, Les Mikesell wrote: > On Fri, Jan 20, 2012 at 9:15 AM, smallpox wrote: >> i was under the impression that rsync does the comparison with little or >> no bandwidth. > It generally does not use a big percentage of the data, but the client > sends the full directory tree data to the server where the comparison > happens. If you have millions of files, that's going to use some > bandwidth even you are doing incrementals and the directory comparison > shows that no timestamps have changed. By contrast, for a tar > incremental the server would send a timestamp and the client would > only send newer files - but that misses deletions, new files that > maintain old timestamps, the new location of old files under a renamed > directory, etc., etc.. > -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] backuppc incremental taking up a lot of bandwidth with no additional changes
On Fri, Jan 20, 2012 at 9:15 AM, smallpox wrote: > i was under the impression that rsync does the comparison with little or > no bandwidth. It generally does not use a big percentage of the data, but the client sends the full directory tree data to the server where the comparison happens. If you have millions of files, that's going to use some bandwidth even you are doing incrementals and the directory comparison shows that no timestamps have changed. By contrast, for a tar incremental the server would send a timestamp and the client would only send newer files - but that misses deletions, new files that maintain old timestamps, the new location of old files under a renamed directory, etc., etc.. -- Les Mikesell lesmikes...@gmail.com -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] backuppc incremental taking up a lot of bandwidth with no additional changes
i was under the impression that rsync does the comparison with little or no bandwidth. On 1/20/2012 5:50 AM, Michael Stowe wrote: >> BackupPC 3.2.1 >> Windows 7 rsyncd over ssh, west coast >> the server is in the east. >> >> my goal is to have it updated every 15 minutes, i've gotten that to work >> but for incremental, with no changes, it's still taking about 10 minutes >> most of the time and it is doing traffic. > I would expect most of that traffic would be checking to see if there are > changes to the files. > >> i do not understand why incremental with absolutely no changes is taking >> so long ? ultimately i'd like to have a hundred thousand files being >> backed up constantly, is this just a dream ? > How would you propose that changes be recognized, if not through rsync's > mechanisms of comparison? > > You can always go to cmd/cifs and use time stamps to determine what's > changed, I suppose. > >> thanks in advance > > -- > Keep Your Developer Skills Current with LearnDevNow! > The most comprehensive online learning library for Microsoft developers > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, > Metro Style Apps, more. Free future releases when you subscribe now! > http://p.sf.net/sfu/learndevnow-d2d > ___ > BackupPC-users mailing list > BackupPC-users@lists.sourceforge.net > List:https://lists.sourceforge.net/lists/listinfo/backuppc-users > Wiki:http://backuppc.wiki.sourceforge.net > Project: http://backuppc.sourceforge.net/ > -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] backuppc incremental taking up a lot of bandwidth with no additional changes
> BackupPC 3.2.1 > Windows 7 rsyncd over ssh, west coast > the server is in the east. > > my goal is to have it updated every 15 minutes, i've gotten that to work > but for incremental, with no changes, it's still taking about 10 minutes > most of the time and it is doing traffic. I would expect most of that traffic would be checking to see if there are changes to the files. > i do not understand why incremental with absolutely no changes is taking > so long ? ultimately i'd like to have a hundred thousand files being > backed up constantly, is this just a dream ? How would you propose that changes be recognized, if not through rsync's mechanisms of comparison? You can always go to cmd/cifs and use time stamps to determine what's changed, I suppose. > thanks in advance -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] backuppc incremental taking up a lot of bandwidth with no additional changes
BackupPC 3.2.1 Windows 7 rsyncd over ssh, west coast the server is in the east. Backup# Type Filled Level Start Date Duration/mins Age/days Server Backup Path 0 full yes 0 1/12 23:47 0.3 6.9 /home/backuppc/BackupPC-3.2.1/pc/danger/0 3 incr no 1 1/14 08:00 12.8 5.6 /home/backuppc/BackupPC-3.2.1/pc/danger/3 4 incr no 1 1/15 08:00 10.7 4.6 /home/backuppc/BackupPC-3.2.1/pc/danger/4 5 incr no 1 1/16 20:00 10.4 3.1 /home/backuppc/BackupPC-3.2.1/pc/danger/5 6 incr no 1 1/17 20:00 3.0 2.1 /home/backuppc/BackupPC-3.2.1/pc/danger/6 7 incr no 1 1/18 20:00 44.6 1.1 /home/backuppc/BackupPC-3.2.1/pc/danger/7 8 incr no 1 1/18 22:30 10.4 1.0 /home/backuppc/BackupPC-3.2.1/pc/danger/8 9 incr no 150 1/19 01:00 0.2 0.9 /home/backuppc/BackupPC-3.2.1/pc/danger/9 10 incr no 150 1/19 03:30 0.2 0.8 /home/backuppc/BackupPC-3.2.1/pc/danger/10 11 incr no 150 1/19 06:00 0.2 0.7 /home/backuppc/BackupPC-3.2.1/pc/danger/11 12 incr no 1 1/19 09:00 10.5 0.6 /home/backuppc/BackupPC-3.2.1/pc/danger/12 13 incr no 1 1/19 09:15 10.5 0.5 /home/backuppc/BackupPC-3.2.1/pc/danger/13 14 incr no 1 1/19 09:30 3.1 0.5 /home/backuppc/BackupPC-3.2.1/pc/danger/14 15 incr no 1 1/19 09:45 10.5 0.5 /home/backuppc/BackupPC-3.2.1/pc/danger/15 16 incr no 1 1/19 10:00 10.6 0.5 /home/backuppc/BackupPC-3.2.1/pc/danger/16 17 incr no 1 1/19 10:15 10.5 0.5 /home/backuppc/BackupPC-3.2.1/pc/danger/17 18 incr no 1 1/19 10:30 3.1 0.5 /home/backuppc/BackupPC-3.2.1/pc/danger/18 19 incr no 1 1/19 10:45 10.5 0.5 /home/backuppc/BackupPC-3.2.1/pc/danger/19 20 incr no 1 1/19 11:00 3.1 0.5 /home/backuppc/BackupPC-3.2.1/pc/danger/20 21 incr no 1 1/19 13:30 10.4 0.4 /home/backuppc/BackupPC-3.2.1/pc/danger/21 22 incr no 1 1/19 13:45 10.5 0.4 /home/backuppc/BackupPC-3.2.1/pc/danger/22 23 incr no 1 1/19 14:00 10.4 0.3 /home/backuppc/BackupPC-3.2.1/pc/danger/23 24 incr no 1 1/19 14:15 3.0 0.3 /home/backuppc/BackupPC-3.2.1/pc/danger/24 my goal is to have it updated every 15 minutes, i've gotten that to work but for incremental, with no changes, it's still taking about 10 minutes most of the time and it is doing traffic. Totals Existing Files New Files Backup# Type #Files Size/MB MB/sec #Files Size/MB #Files Size/MB 0 full 137 4.0 0.25 19 0.0 140 4.0 3 incr 4 384.6 0.50 4 384.6 0 0.0 4 incr 4 384.6 0.60 4 384.6 0 0.0 5 incr 4 384.6 0.62 4 384.6 0 0.0 6 incr 0 0.0 0.00 0 0.0 1 0.0 7 incr 4 384.6 0.14 4 384.6 4 0.0 8 incr 4 384.6 0.61 4 384.6 0 0.0 9 incr 0 0.0 0.00 0 0.0 0 0.0 10 incr 0 0.0 0.00 0 0.0 0 0.0 11 incr 0 0.0 0.00 0 0.0 0 0.0 12 incr 4 384.6 0.61 4 384.6 0 0.0 13 incr 4 384.6 0.61 4 384.6 0 0.0 14 incr 0 0.0 0.00 0 0.0 0 0.0 15 incr 4 384.6 0.61 4 384.6 0 0.0 16 incr 4 384.6 0.61 4 384.6 0 0.0 17 incr 4 384.6 0.61 4 384.6 0 0.0 18 incr 0 0.0 0.00 0 0.0 0 0.0 19 incr 4 384.6 0.61 4 384.6 0 0.0 20 incr 0 0.0 0.00 0 0.0 0 0.0 21 incr 4 384.6 0.62 4 384.6 0 0.0 22 incr 4 384.6 0.61 4 384.6 0 0.0 23 incr 4 384.6 0.61 4 384.6 0 0.0 24 incr 0 0.0 0.00 0 0.0 0 0.0 i do not understand why incremental with absolutely no changes is taking so long ? ultimately i'd like to have a hundred thousand files being backed up constantly, is this just a dream ? thanks in advance -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sf
