Because this is a dump, a small change in a record near the beginning of the
sort would cause the whole file to be a different structure. If you can
influence the sort process for the dump and do it by date that the record
was updated then you wouldnt likely have this issue.
example (csv formated)
1,aerthdfd,test,1/1/2006
2,asdfsad,stuff,3/6/2009
3,agrtdhh,data,4/16/2009
if you dump this once you get a file with:
1,aerthdfd,test,1/1/2006
2,asdfsad,stuff,3/6/2009
3,agrtdhh,data,4/16/2009
if you add two more data rows
4,aaewrr,junk,6/7/2009
5,atiojojj,last,6/8/2009
your export might be
4,aaewrr,junk,6/7/2009
1,aerthdfd,test,1/1/2006
2,asdfsad,stuff,3/6/2009
5,atiojojj,last,6/8/2009
3,agrtdhh,data,4/16/2009
which rsync would match up because the data has changed location within the
file. if you can sort by the row id, or the date, rsync will be much
happier
On Tue, Aug 25, 2009 at 6:37 AM, Les Mikesell <lesmikes...@gmail.com> wrote:
> Nigel Kendrick wrote:
> > Hi,
> >
> > This may also be applicable to MySQL etc..
> >
> > I have an MS-SQL database that dumps out a backup that's around 700MB in
> > size. BackupPC brought this over to its server via ADSL/VPN/Rsyncd
> > (422mins!) and when I did another data dump about 4 hours later, around
> > 16Mb of changes to the file were transferred very quickly. However,
> > every single backup since then has gone back to 422 minutes. I could
> > imagine that the database and dump will change over time, but it seems
> > strange that the entire file needs copying every time - as if its
> > structure is always 100% different from the previous.
> >
> > I wondered if I was missing anything obvious (being a BackupPC noob)
> > like file time stamping causing a complete file transfer every day, or
> > are database dumps likely to be completely different every time?
> >
> > I will keep some copies of the dumps and do some comparisons but in the
> > meantime any tips, thoughts etc.?
>
> I don't know anything about the contents of the file, but are these full or
> incremental runs for backuppc? Assuming this is rsync or rsyncd, note
> that
> incrementals always compare against the last full unless you have
> configured
> multiple incremental levels, so they tend to transfer more each time. Or
> there
> may just be enough difference that it can't find the matching parts.
>
> If you are running rsync with ssh (probably not if this is windows), you
> might
> save some time by adding the -C option for ssh compression. If transfer
> time is
> important, it might be worth setting up a scheduled rsync command to copy
> to
> something on the network local to the backuppc server so you could use the
> -z
> option (which backuppc doesn't support), then have backuppc pick up that
> copy to
> keep the history. Or, if rsync isn't finding matching parts anyway perhaps
> you
> could compress the file before the transfer.
>
> --
> Les Mikesell
> lesmikes...@gmail.com
>
>
> ------------------------------------------------------------------------------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
> trial. Simplify your report design, integration and deployment - and focus
> on
> what you do best, core application coding. Discover what's new with
> Crystal Reports now. http://p.sf.net/sfu/bobj-july
> _______________________________________________
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki: http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
>
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
