Re: [BackupPC-users] why are full backups needed with BackupPC?
On Tue, 2006-28-03 at 15:00 -0600, Les Mikesell wrote: > On Tue, 2006-03-28 at 13:44, Kanwar Ranbir Sandhu wrote: > > > What hasn't worked for, ever, is using ssh to connect to the Windows > > box, and then pull the backups down by connecting to the rsyncd. That > > being said, I seem to recall someone reporting that they got it to work, > > so I should probably take a look at the list archives. > > > > Anyway, have you tried this? Has it worked for? Holy shit - what a horrible reply. I'm half asleep...guess coherence goes out the window when I type with me eyes half closed. :) Sorry about all the missing words! > I haven't done it with backuppc, but it should work using > ssh only as a port-forwarding tunnel. The only problem is > that you don't know when the backup is complete so you > might have to leave the ssh connection up most of the time. Yeah, that much I know. Perhaps I didn't do the port forwarding correctly the last time I tried this. The other problem of course was with sites that have multiple servers behind a firewall. That would mean poking a few holes through a firewall, and opening SSH up. I know access can be restricted, but a lot of sites just don't like it. > I have VPN connections (currently CIPE but I'd probably use > openvpn now) to all of the places where I need backups so > I haven't worked very hard at ssh'ing into windows. But, > I've been hoping someone would track down the deadlock since > about everything else works with cygwin sshd. That's what I ended up doing. OpenVPN for all the clients that are either running Windows or don't want to open up SSH, and SSH for those Linux sites that don't mind. SSH is by far the simplest and fastest setup. Regards, Ranbir -- Kanwar Ranbir Sandhu Linux 2.6.15-1.1833_FC4 i686 GNU/Linux 15:57:03 up 9:26, 3 users, load average: 0.31, 0.50, 0.36 --- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] why are full backups needed with BackupPC?
On Tue, 2006-03-28 at 13:44, Kanwar Ranbir Sandhu wrote: > What hasn't worked for, ever, is using ssh to connect to the Windows > box, and then pull the backups down by connecting to the rsyncd. That > being said, I seem to recall someone reporting that they got it to work, > so I should probably take a look at the list archives. > > Anyway, have you tried this? Has it worked for? I haven't done it with backuppc, but it should work using ssh only as a port-forwarding tunnel. The only problem is that you don't know when the backup is complete so you might have to leave the ssh connection up most of the time. I have VPN connections (currently CIPE but I'd probably use openvpn now) to all of the places where I need backups so I haven't worked very hard at ssh'ing into windows. But, I've been hoping someone would track down the deadlock since about everything else works with cygwin sshd. -- Les Mikesell [EMAIL PROTECTED] --- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] why are full backups needed with BackupPC?
On Tue, 2006-28-03 at 12:14 -0600, Les Mikesell wrote: > Rsyncd is rsync listening directly in --daemon mode instead of > being started by sshd. You shouldn't have any trouble setting > that up under windows except possibly making it run as a > service. That bit works, though once in a while the rsyncd service (on Windows) decides to die once in a while. What hasn't worked for, ever, is using ssh to connect to the Windows box, and then pull the backups down by connecting to the rsyncd. That being said, I seem to recall someone reporting that they got it to work, so I should probably take a look at the list archives. Anyway, have you tried this? Has it worked for? Regards, Ranbir -- Kanwar Ranbir Sandhu Linux 2.6.15-1.1833_FC4 i686 GNU/Linux 14:41:24 up 8:11, 3 users, load average: 0.57, 0.39, 0.34 --- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] why are full backups needed with BackupPC?
On Tue, 2006-03-28 at 10:50, Kanwar Ranbir Sandhu wrote: > On Mon, 2006-27-03 at 22:11 -0600, Trey Nolen wrote: > > Rsyncd over ssh work on windows...rsync will still not act like a server > > under cygwin. > > I've tried using rsyncd with cygwin on the Windows side - it's never > worked for me. How did you get it to work? (or am I misunderstanding > something?) Rsyncd is rsync listening directly in --daemon mode instead of being started by sshd. You shouldn't have any trouble setting that up under windows except possibly making it run as a service. -- Les Mikesell [EMAIL PROTECTED] --- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] why are full backups needed with BackupPC?
On Mon, 2006-27-03 at 22:11 -0600, Trey Nolen wrote: > Rsyncd over ssh work on windows...rsync will still not act like a server > under cygwin. I've tried using rsyncd with cygwin on the Windows side - it's never worked for me. How did you get it to work? (or am I misunderstanding something?) Regards, Ranbir -- Kanwar Ranbir Sandhu Linux 2.6.15-1.1833_FC4 i686 GNU/Linux 11:48:59 up 5:18, 2 users, load average: 0.11, 0.15, 0.16 --- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] why are full backups needed with BackupPC?
Rsyncd over ssh work on windows...rsync will still not act like a server under cygwin. Trey - Original Message - From: "Dan Pritts" <[EMAIL PROTECTED]> To: "Laurent Mazet" <[EMAIL PROTECTED]> Cc: Sent: Monday, March 27, 2006 10:02 PM Subject: Re: [BackupPC-users] why are full backups needed with BackupPC? On Mon, Mar 27, 2006 at 12:52:31AM +0200, Laurent Mazet wrote: To summarize, for a Windows host: - rsync over ssh doesn't work. - rsyncd transfers only diff but you need to connect with a clear password. You might be able to run rsyncd transfers over an ssh tunnel. I don't know if there are issues with that on cygwin or not. Does anyone know if rsync and ssh will run under uwin? or if uwin still even really works? danno -- dan pritts - systems administrator - internet2 734/352-4953 office734/834-7224 mobile --- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/ --- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] why are full backups needed with BackupPC?
On Mon, Mar 27, 2006 at 12:52:31AM +0200, Laurent Mazet wrote: > To summarize, for a Windows host: > - rsync over ssh doesn't work. > - rsyncd transfers only diff but you need to connect with a clear password. You might be able to run rsyncd transfers over an ssh tunnel. I don't know if there are issues with that on cygwin or not. Does anyone know if rsync and ssh will run under uwin? or if uwin still even really works? danno -- dan pritts - systems administrator - internet2 734/352-4953 office734/834-7224 mobile --- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] why are full backups needed with BackupPC?
On Sun, 26 Mar 2006 15:09:06 -0800 Craig Barratt <[EMAIL PROTECTED]> wrote: > Laurent Mazet writes: > > > To summarize, for a Windows host: > > > > - rsync over ssh doesn't work. > > Yes, but I haven't tested it recently. Unfortunately, I've tried last week with copSSH 1.3.9 and cwRsync 2.0.6 from http://www.itefix.no without any success > > - rsyncd transfers only diff but you need to connect with a clear password. > > Rsyncd doesn't send a clear password over the network. It uses > a digest-based challenge/response. At least you need you need to store the password in a plain text > > - tar over ssh transfers every thing. > > Yes, for a full. > > > - smb transfers every thing and you need to connect with a clear password. > > I'm not sure whether smb sends clear passwords over the network. I thing your right. It might depend on a boolean (encrypt passwords) in samba config. But you need to store it in an a plain text and it also gives a full login access. For security reason, I thing that the worst method. Laurent --- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] why are full backups needed with BackupPC?
Laurent Mazet writes: > To summarize, for a Windows host: > > - rsync over ssh doesn't work. Yes, but I haven't tested it recently. > - rsyncd transfers only diff but you need to connect with a clear password. Rsyncd doesn't send a clear password over the network. It uses a digest-based challenge/response. > - tar over ssh transfers every thing. Yes, for a full. > - smb transfers every thing and you need to connect with a clear password. I'm not sure whether smb sends clear passwords over the network. Craig --- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid0944&bid$1720&dat1642 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] why are full backups needed with BackupPC?
On Sun, 26 Mar 2006 09:19:02 -0500 Dan Pritts <[EMAIL PROTECTED]> wrote: > On Sat, Mar 25, 2006 at 02:48:31PM +0100, Tomasz Chmielewski wrote: > > What about "smb" method for Windows? Will full backup transfer > > everything (as tar would), or will it do some magic and transfer > > new/changed files only (based on timestamps etc.)? > > it transfers everything. > > > If it transfers everything, perhaps it's better to install rsynd on a > > Windows machine? > > perhaps. depends on whether that additional complexity is worth it to > you. If you are bandwidth-constrained, then probably better. If it's > on a LAN, maybe not worth the hassle. To summarize, for a Windows host: - rsync over ssh doesn't work. - rsyncd transfers only diff but you need to connect with a clear password. - tar over ssh transfers every thing. - smb transfers every thing and you need to connect with a clear password. Correct me if I'm wrong. --- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] why are full backups needed with BackupPC?
On Sun, 2006-03-26 at 11:10, Dan Young wrote: > > > > Rsync will read everything at both ends during a full which > > can be fairly slow, but only a small amount of bandwidth > > is used for the comparison and otherwise only the changes > > are sent. > > Unless you turn on checksum caching, correct? Then the BackupPC side > will not recompute the checksums for files in the pool, except for at > some defined interval (randomly on 1% of the pool by default). Yes, but the other end still has to read every file so it is still a lot slower than it would be if it skipped files when the timestamp and length are still the same. -- Les Mikesell [EMAIL PROTECTED] --- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] why are full backups needed with BackupPC?
On Mar 25, 2006, at 8:53 AM, Les Mikesell wrote: On Sat, 2006-03-25 at 02:45, Tomasz Chmielewski wrote: Does the full rsync backup in BackupPC transfer only changes (compared to the last full backup), or maybe it transfers everything? It's not clear from the documentation (which states: "A full backup is a complete backup of a share."). Rsync will read everything at both ends during a full which can be fairly slow, but only a small amount of bandwidth is used for the comparison and otherwise only the changes are sent. Unless you turn on checksum caching, correct? Then the BackupPC side will not recompute the checksums for files in the pool, except for at some defined interval (randomly on 1% of the pool by default). -- Dan Young <[EMAIL PROTECTED]> Multnomah ESD - Technology Services 503-257-1562 --- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] why are full backups needed with BackupPC?
On Sat, Mar 25, 2006 at 02:48:31PM +0100, Tomasz Chmielewski wrote: > What about "smb" method for Windows? Will full backup transfer > everything (as tar would), or will it do some magic and transfer > new/changed files only (based on timestamps etc.)? it transfers everything. > If it transfers everything, perhaps it's better to install rsynd on a > Windows machine? perhaps. depends on whether that additional complexity is worth it to you. If you are bandwidth-constrained, then probably better. If it's on a LAN, maybe not worth the hassle. danno -- dan pritts - systems administrator - internet2 734/352-4953 office734/834-7224 mobile --- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] why are full backups needed with BackupPC?
On Sat, 2006-03-25 at 02:45, Tomasz Chmielewski wrote: > > > > You should be able to tell backuppc to make fulls as often as > > you want. The only downside with rsync is the extra time > > it takes to do the full block checksum compare on existing > > files. > > Is it really the only downside of full backups? > > Doesn't a full backup mean that *everything* will be transferred again? The tar/smb methods would transfer everything. Rsync will compare everything with it's block checksum technique. > In case of backup of several 50 GB servers over slow internet, that > would be a never-ending daily disaster :) > > > Does the full rsync backup in BackupPC transfer only changes (compared > to the last full backup), or maybe it transfers everything? > It's not clear from the documentation (which states: "A full backup is a > complete backup of a share."). Rsync will read everything at both ends during a full which can be fairly slow, but only a small amount of bandwidth is used for the comparison and otherwise only the changes are sent. In the case of a slightly modified file like a growing logfile, rsync will reconstruct it at the receiving end from the part already their plus the differences sent. This will result in a new copy in the pool. In the case where an existing file is updated in the same way on several targets, the new duplicates will be identified and linked in the pool. If you wanted to fiddle with the configs frequently you could probably always run fulls, but change the command for rsync fulls so on weekdays the timestamp/length match is enough to skip a transfer but on weekends you do the full compares. -- Les Mikesell [EMAIL PROTECTED] --- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] why are full backups needed with BackupPC?
Paul Fox wrote: > > > > You should be able to tell backuppc to make fulls as often as > > you want. The only downside with rsync is the extra time > > it takes to do the full block checksum compare on existing > > files. > > Is it really the only downside of full backups? > > Doesn't a full backup mean that *everything* will be transferred again? with tar, yes. with rsync, no -- rsync only recompares checksums. All right, than perhaps it's good to make only "full" backups with rsync/rsyncd, and no incremental backups. What about "smb" method for Windows? Will full backup transfer everything (as tar would), or will it do some magic and transfer new/changed files only (based on timestamps etc.)? If it transfers everything, perhaps it's better to install rsynd on a Windows machine? -- Tomasz Chmielewski http://wpkg.org --- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] why are full backups needed with BackupPC?
> > > > You should be able to tell backuppc to make fulls as often as > > you want. The only downside with rsync is the extra time > > it takes to do the full block checksum compare on existing > > files. > > Is it really the only downside of full backups? > > Doesn't a full backup mean that *everything* will be transferred again? with tar, yes. with rsync, no -- rsync only recompares checksums. > Does the full rsync backup in BackupPC transfer only changes (compared > to the last full backup), or maybe it transfers everything? it ignores any hints that are used during incrementals (dates, modes, etc), and transfers everything that isn't already available in the pool. the result is the same as if it transferred everything. paul =- paul fox, [EMAIL PROTECTED] (arlington, ma, where it's 34.7 degrees) --- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] why are full backups needed with BackupPC?
Les Mikesell wrote: On Fri, 2006-03-24 at 17:21, Tomasz Chmielewski wrote: there are other backup schemes that do the "always make a full" thing, do a web search for "mike rubel rsync backup" to see one guy's web page that uses a similar scheme. It lacks many of the nice surrounding tools that backuppc gives you. I'd rather want BackupPC to have it :) You should be able to tell backuppc to make fulls as often as you want. The only downside with rsync is the extra time it takes to do the full block checksum compare on existing files. Is it really the only downside of full backups? Doesn't a full backup mean that *everything* will be transferred again? In case of backup of several 50 GB servers over slow internet, that would be a never-ending daily disaster :) Does the full rsync backup in BackupPC transfer only changes (compared to the last full backup), or maybe it transfers everything? It's not clear from the documentation (which states: "A full backup is a complete backup of a share."). -- Tomasz Chmielewski http://wpkg.org --- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] why are full backups needed with BackupPC?
On Fri, 2006-03-24 at 17:56, Peter Gumeson wrote: > After looking through the backuppc code, I am under > the impression that BackupPC does not take advantage > of using the --link-dest option in Rsync to create > hardlinks. I am guessing one reason may be that the > PerlRsync module does not appear to support the > --link-dest option. (http://perlrsync.sourceforge.net) Backuppc does its own hardlinking more aggressively than rsync can. Any duplicate files in the backuppc archive will end up being hard links to a common pooled copy regardles of whether the duplicates are from subsequent backup runs of the same location, duplicate files on the same machine, or from different machines entirely. The best rsync can do is to link to a file from the same place from the same machine in a previous run. -- Les Mikesell [EMAIL PROTECTED] --- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] why are full backups needed with BackupPC?
Doh! I see what you meant, you suggested adding link_dest to the rsync args in the config.pl Is this safe with backuppc? Peter --- Peter Gumeson <[EMAIL PROTECTED]> wrote: > > Hi all, > > After looking through the backuppc code, I am under > the impression that BackupPC does not take advantage > of using the --link-dest option in Rsync to create > hardlinks. I am guessing one reason may be that the > PerlRsync module does not appear to support the > --link-dest option. > (http://perlrsync.sourceforge.net) > > One project that I know uses link_dest is Rsnapshot, > and the advantages of using link_dest are described > as > follows: > > --- > > 4.3.6. link_dest > > If you have rsync version 2.5.7 or later, you may > want > to enable this. With link_dest enabled, rsnapshot > relies on rsync to create recursive hard links, > overriding GNU cp in most, but not all, cases. With > link_dest enabled, every single file on your system > can be backed up in one pass, on any operating > system. > To get the most out of rsnapshot on non-Linux > platforms, link_dest should be enabled. Be advised, > however, that if a remote host is unavailable during > a > backup, rsnapshot will take an extra step and roll > back the files from the previous backup. Using GNU > cp, > this would not be necessary. > > http://www.rsnapshot.org/howto/1.2/rsnapshot-HOWTO.en.html > --- > > However, BackupPC seems to be a much more mature > project and has many more out of the box features > and > built-in OS compatibility. I am wondering if there > were some initial limitations that prevented Craig > from using link-dest. Or possibly I am missing > something and it's in there? Does anybody know? > > Peter Gumeson > > > > --- Tomasz Chmielewski <[EMAIL PROTECTED]> wrote: > > > Dan Pritts wrote: > > > On Fri, Mar 24, 2006 at 05:35:12PM +0100, Tomasz > > Chmielewski wrote: > > >> Why are full backups needed at all with > BackupPC? > > >> > > >> According to documentation, "BackupPC's CGI > > interface ``fills-in'' > > >> incremental backups based on the last full > > backup, giving every backup a > > >> ``full'' appearance." > > >> > > >> So, in theory, it should be enough to make just > > one full, initial > > >> backup, and then only incremental backups. > > > > > > You can do that but as things change on the > backup > > client, you will end > > > up with larger and larger incremental backups, > > using more bandwidth on > > > the wire. Maybe an issue for you, maybe not. > > > > This is why it wonders me. > > So far I've been using my own scripts based on > > rsync. > > > > Basically, what I did, was to create a "latest" > dir, > > and then hardlink > > each and every file from it, to a given day dir: > > > > latest - 2006-03-25 > > - 2006-03-24 > > - 2006-03-23 > > > > etc. > > > > This way I had the exact state of the machines > which > > were backed up with > > rsync at any given day. > > > > I decided to use BackupPC, as it features > > compression (important, as > > there is no compressed filesystem for Linux), and > > pooling (less > > important, but why not). > > > > > > > I have read on this list that when you use > rsync, > > your "full" backups > > > won't transfer everything over the wire, but > > rather will intelligently > > > make use of the previous full, so there's not a > > big loss to having a full. > > > > Yes, just use --link-dest option when hardlinking. > > > > > > > one issue you might need to weigh is that each > > full means lots and > > > lots of hard links get created on your > filesystem. > > this takes lots > > > of time since the disk heads have to seek all > over > > the place. > > > On the other hand, incrementals won't do this. > > > > No, it doesn't take that long, hardlinking one > > server (say 5 GB data) to > > another directory will take only a few minutes. > > > > > > > there are other backup schemes that do the > "always > > make a full" thing, > > > do a web search for "mike rubel rsync backup" to > > see one guy's web > > > page that uses a similar scheme. It lacks many > of > > the nice surrounding > > > tools that backuppc gives you. > > > > I'd rather want BackupPC to have it :) > > > > > > -- > > Tomasz Chmielewski > > http://wpkg.org > > > > > > > --- > > This SF.Net email is sponsored by xPML, a > > groundbreaking scripting language > > that extends applications into web and mobile > media. > > Attend the live webcast > > and join the prime developer group breaking into > > this new coding territory! > > > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > > ___ > > BackupPC-users mailing list > > BackupPC-users@lists.sourceforge.net > > > https://lists.sourceforge.net/lists/listinfo/backuppc-users > > http://backuppc.sourceforge.net/ > > > > > __
Re: [BackupPC-users] why are full backups needed with BackupPC?
Hi all, After looking through the backuppc code, I am under the impression that BackupPC does not take advantage of using the --link-dest option in Rsync to create hardlinks. I am guessing one reason may be that the PerlRsync module does not appear to support the --link-dest option. (http://perlrsync.sourceforge.net) One project that I know uses link_dest is Rsnapshot, and the advantages of using link_dest are described as follows: --- 4.3.6. link_dest If you have rsync version 2.5.7 or later, you may want to enable this. With link_dest enabled, rsnapshot relies on rsync to create recursive hard links, overriding GNU cp in most, but not all, cases. With link_dest enabled, every single file on your system can be backed up in one pass, on any operating system. To get the most out of rsnapshot on non-Linux platforms, link_dest should be enabled. Be advised, however, that if a remote host is unavailable during a backup, rsnapshot will take an extra step and roll back the files from the previous backup. Using GNU cp, this would not be necessary. http://www.rsnapshot.org/howto/1.2/rsnapshot-HOWTO.en.html --- However, BackupPC seems to be a much more mature project and has many more out of the box features and built-in OS compatibility. I am wondering if there were some initial limitations that prevented Craig from using link-dest. Or possibly I am missing something and it's in there? Does anybody know? Peter Gumeson --- Tomasz Chmielewski <[EMAIL PROTECTED]> wrote: > Dan Pritts wrote: > > On Fri, Mar 24, 2006 at 05:35:12PM +0100, Tomasz > Chmielewski wrote: > >> Why are full backups needed at all with BackupPC? > >> > >> According to documentation, "BackupPC's CGI > interface ``fills-in'' > >> incremental backups based on the last full > backup, giving every backup a > >> ``full'' appearance." > >> > >> So, in theory, it should be enough to make just > one full, initial > >> backup, and then only incremental backups. > > > > You can do that but as things change on the backup > client, you will end > > up with larger and larger incremental backups, > using more bandwidth on > > the wire. Maybe an issue for you, maybe not. > > This is why it wonders me. > So far I've been using my own scripts based on > rsync. > > Basically, what I did, was to create a "latest" dir, > and then hardlink > each and every file from it, to a given day dir: > > latest - 2006-03-25 > - 2006-03-24 > - 2006-03-23 > > etc. > > This way I had the exact state of the machines which > were backed up with > rsync at any given day. > > I decided to use BackupPC, as it features > compression (important, as > there is no compressed filesystem for Linux), and > pooling (less > important, but why not). > > > > I have read on this list that when you use rsync, > your "full" backups > > won't transfer everything over the wire, but > rather will intelligently > > make use of the previous full, so there's not a > big loss to having a full. > > Yes, just use --link-dest option when hardlinking. > > > > one issue you might need to weigh is that each > full means lots and > > lots of hard links get created on your filesystem. > this takes lots > > of time since the disk heads have to seek all over > the place. > > On the other hand, incrementals won't do this. > > No, it doesn't take that long, hardlinking one > server (say 5 GB data) to > another directory will take only a few minutes. > > > > there are other backup schemes that do the "always > make a full" thing, > > do a web search for "mike rubel rsync backup" to > see one guy's web > > page that uses a similar scheme. It lacks many of > the nice surrounding > > tools that backuppc gives you. > > I'd rather want BackupPC to have it :) > > > -- > Tomasz Chmielewski > http://wpkg.org > > > --- > This SF.Net email is sponsored by xPML, a > groundbreaking scripting language > that extends applications into web and mobile media. > Attend the live webcast > and join the prime developer group breaking into > this new coding territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > ___ > BackupPC-users mailing list > BackupPC-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/backuppc-users > http://backuppc.sourceforge.net/ > __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com --- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
Re: [BackupPC-users] why are full backups needed with BackupPC?
On Fri, 2006-03-24 at 17:21, Tomasz Chmielewski wrote: > > there are other backup schemes that do the "always make a full" thing, > > do a web search for "mike rubel rsync backup" to see one guy's web > > page that uses a similar scheme. It lacks many of the nice surrounding > > tools that backuppc gives you. > > I'd rather want BackupPC to have it :) You should be able to tell backuppc to make fulls as often as you want. The only downside with rsync is the extra time it takes to do the full block checksum compare on existing files. -- Les Mikesell [EMAIL PROTECTED] --- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] why are full backups needed with BackupPC?
Dan Pritts wrote: On Fri, Mar 24, 2006 at 05:35:12PM +0100, Tomasz Chmielewski wrote: Why are full backups needed at all with BackupPC? According to documentation, "BackupPC's CGI interface ``fills-in'' incremental backups based on the last full backup, giving every backup a ``full'' appearance." So, in theory, it should be enough to make just one full, initial backup, and then only incremental backups. You can do that but as things change on the backup client, you will end up with larger and larger incremental backups, using more bandwidth on the wire. Maybe an issue for you, maybe not. This is why it wonders me. So far I've been using my own scripts based on rsync. Basically, what I did, was to create a "latest" dir, and then hardlink each and every file from it, to a given day dir: latest - 2006-03-25 - 2006-03-24 - 2006-03-23 etc. This way I had the exact state of the machines which were backed up with rsync at any given day. I decided to use BackupPC, as it features compression (important, as there is no compressed filesystem for Linux), and pooling (less important, but why not). I have read on this list that when you use rsync, your "full" backups won't transfer everything over the wire, but rather will intelligently make use of the previous full, so there's not a big loss to having a full. Yes, just use --link-dest option when hardlinking. one issue you might need to weigh is that each full means lots and lots of hard links get created on your filesystem. this takes lots of time since the disk heads have to seek all over the place. On the other hand, incrementals won't do this. No, it doesn't take that long, hardlinking one server (say 5 GB data) to another directory will take only a few minutes. there are other backup schemes that do the "always make a full" thing, do a web search for "mike rubel rsync backup" to see one guy's web page that uses a similar scheme. It lacks many of the nice surrounding tools that backuppc gives you. I'd rather want BackupPC to have it :) -- Tomasz Chmielewski http://wpkg.org --- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] why are full backups needed with BackupPC?
On Fri, Mar 24, 2006 at 05:35:12PM +0100, Tomasz Chmielewski wrote: > Why are full backups needed at all with BackupPC? > > According to documentation, "BackupPC's CGI interface ``fills-in'' > incremental backups based on the last full backup, giving every backup a > ``full'' appearance." > > So, in theory, it should be enough to make just one full, initial > backup, and then only incremental backups. You can do that but as things change on the backup client, you will end up with larger and larger incremental backups, using more bandwidth on the wire. Maybe an issue for you, maybe not. a I have read on this list that when you use rsync, your "full" backups won't transfer everything over the wire, but rather will intelligently make use of the previous full, so there's not a big loss to having a full. one issue you might need to weigh is that each full means lots and lots of hard links get created on your filesystem. this takes lots of time since the disk heads have to seek all over the place. On the other hand, incrementals won't do this. there are other backup schemes that do the "always make a full" thing, do a web search for "mike rubel rsync backup" to see one guy's web page that uses a similar scheme. It lacks many of the nice surrounding tools that backuppc gives you. danno -- dan pritts - systems administrator - internet2 734/352-4953 office734/834-7224 mobile --- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] why are full backups needed with BackupPC?
On Fri, 2006-03-24 at 11:04, Tomasz Chmielewski wrote: > > > So, in theory, it should be enough to make just one full, initial > > > backup, and then only incremental backups. > > > > > > Or do I miss something here? > > > > one reason (there may be others) is that incrementals don't account > > for the removal of files. if a full contains a file that is later > > removed, it will always appear in that "filled" view, even after > > the file is gone from your system. so full backups are necessary > > to reestablish a true image of your current contents. > > > > (this is with tar -- rsync incrementals may actually remove deleted > > files. i don't use rsync.) > > So with rsync it shouldn't be an issue, right? Could anyone comment on that? They have different issues. I'm not sure about that particular one. Tar/smb backups use only the file timestamps to decide what to take in incremental runs and will thus miss files in a new location because a directory above was renamed and in at least the smb case, files that are back-dated by copy mechanisms that preserve the original timestamp (unzip, etc.). These omissions accumulate until you do the next full. Rsync can always identify new or moved files compared to your backup, but it will always tranfer the changes made since the last full, so the size will keep increasing as you repeat incremental runs. Rsync also bases comparisons on the file size/timestamp during incremental runs but does block checksum comparisons of everything for fulls. In all cases, identical files are collapsed into hard links to the pool copy, so the only down side of full runs is the extra bandwidth for the tar/smb copies and extra time for rsync. -- Les Mikesell [EMAIL PROTECTED] --- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] why are full backups needed with BackupPC?
> > > Why are full backups needed at all with BackupPC? > > > > > > According to documentation, "BackupPC's CGI interface ``fills-in'' > > > incremental backups based on the last full backup, giving every backup > > a > > > ``full'' appearance." > > > > > > So, in theory, it should be enough to make just one full, initial > > > backup, and then only incremental backups. > > > > > > Or do I miss something here? > > > > one reason (there may be others) is that incrementals don't account > > for the removal of files. if a full contains a file that is later > > removed, it will always appear in that "filled" view, even after > > the file is gone from your system. so full backups are necessary > > to reestablish a true image of your current contents. > > > > (this is with tar -- rsync incrementals may actually remove deleted > > files. i don't use rsync.) > > So with rsync it shouldn't be an issue, right? Could anyone comment on that? > > > Anyway, will several full backups use only one hardlinked file in the > pool, or do full backups use separate, non-hardlinked files? they use only one hardlinked file -- i.e., no extra space. paul =- paul fox, [EMAIL PROTECTED] (arlington, ma, where it's 44.1 degrees) --- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] why are full backups needed with BackupPC?
Paul Fox wrote: > Why are full backups needed at all with BackupPC? > > According to documentation, "BackupPC's CGI interface ``fills-in'' > incremental backups based on the last full backup, giving every backup a > ``full'' appearance." > > So, in theory, it should be enough to make just one full, initial > backup, and then only incremental backups. > > Or do I miss something here? one reason (there may be others) is that incrementals don't account for the removal of files. if a full contains a file that is later removed, it will always appear in that "filled" view, even after the file is gone from your system. so full backups are necessary to reestablish a true image of your current contents. (this is with tar -- rsync incrementals may actually remove deleted files. i don't use rsync.) So with rsync it shouldn't be an issue, right? Could anyone comment on that? Anyway, will several full backups use only one hardlinked file in the pool, or do full backups use separate, non-hardlinked files? -- Tomasz Chmielewski http://wpkg.org --- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] why are full backups needed with BackupPC?
> Why are full backups needed at all with BackupPC? > > According to documentation, "BackupPC's CGI interface ``fills-in'' > incremental backups based on the last full backup, giving every backup a > ``full'' appearance." > > So, in theory, it should be enough to make just one full, initial > backup, and then only incremental backups. > > Or do I miss something here? one reason (there may be others) is that incrementals don't account for the removal of files. if a full contains a file that is later removed, it will always appear in that "filled" view, even after the file is gone from your system. so full backups are necessary to reestablish a true image of your current contents. (this is with tar -- rsync incrementals may actually remove deleted files. i don't use rsync.) paul =- paul fox, [EMAIL PROTECTED] (arlington, ma, where it's 46.0 degrees) --- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
[BackupPC-users] why are full backups needed with BackupPC?
Why are full backups needed at all with BackupPC? According to documentation, "BackupPC's CGI interface ``fills-in'' incremental backups based on the last full backup, giving every backup a ``full'' appearance." So, in theory, it should be enough to make just one full, initial backup, and then only incremental backups. Or do I miss something here? -- Tomasz Chmielewski Software deployment with Samba http://wpkg.org --- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
