Hi Adam, Thanks for the quick response but I doubt I can split 16TB of data and then backup in a timely manner. But it's worth a try. Cheers, Pedro
On Friday 04 March 2011 13:27:13 Adam Goryachev wrote: > On 04/03/11 23:32, Pedro M. S. Oliveira wrote: > > I'm not sure about doing the 16TB (performance, backup duration) so I > > thinking in some kind of block device backup. > > Idea: > > 1 - Create lvm snapshot of the block device > > 2 - Backup lvm snapshot (I could use DD, but then it would be a full backup > > every time I do a backup), something like rsync where the only the changed > > blocks of the block device. > > > > Benefits: > > 1 - Performance, althoughtthe gains only show after 70% of full disk, > > 45%-50% full disk for small files. > > 2 - Restore backup directly into volume. > > 3 - Possibility of mount on a loop device. > > > > Conns: > > The first backup should take ages, and initial FS should have zeros on it's > > free space (so the initial backup can use the compression efficiently) > > This approach is only possible on unix/linux FS. > > The LVM methods for creating snapshots aren't standard and partitioning / > > volume creation need to be addressed and thought before deployment (is this > > a conn??) > > > > The recover method should be able to restore the block device (in this case > > an LVM volume). > > I can see lots of difficulties with this approach but the benefits can be > > great too. > > What do you all think about this. > > Been there, done that, and it works well already (sort of)... > > I have a MS Windows VM which runs under Linux and it's 'hard drive' is > in effect a file. I used to: > 1) use LVM to take a snapshot > 2) copy the raw snapshot image to another location on the same disk > 3) delete the snapshot > 4) split the copy of the image to individual 20M chunks (split) > 5) use backuppc/rsync to backup the chunks > > The problem with this is the time for the backup to complete (about > hours for steps 1 - 3, and another 1 hour for step 4 > > Recently, I skipped step 1 and 3 and just shut down the machine before > taking the copy, this finishes in about 30 minutes now. In any case, > backuppc handles this quite well. > > The reasons for splitting the image are: > 1) backuppc can take better advantage of pooling since most chunks have > not changed (one big file means the entire file changes every backup). > 2) backuppc doesn't seem to handle really large files with changes in > them (ie, performance wise it seems to slow things down a lot). > > Hope this helps... > > PS, I also use the same idea for certain niche applications that produce > a very large 'backup' file, I split it into chunks before letting > backuppc back it up. I also ensure they are de-compressed first, letting > rsync/ssh/backuppc handle the compression at the transport and file > system levels. > > Regards, > Adam > > -- > Adam Goryachev > Website Managers > www.websitemanagers.com.au > > ------------------------------------------------------------------------------ > What You Don't Know About Data Connectivity CAN Hurt You > This paper provides an overview of data connectivity, details > its effect on application quality, and explores various alternative > solutions. http://p.sf.net/sfu/progress-d2d > _______________________________________________ > BackupPC-users mailing list > [email protected] > List: https://lists.sourceforge.net/lists/listinfo/backuppc-users > Wiki: http://backuppc.wiki.sourceforge.net > Project: http://backuppc.sourceforge.net/ > -- ---------------------------------------------------------------------------------------------------------- Pedro M. S. Oliveira Pólo Tecnológico de Lisboa IT Consultant Estr. do Paço do Lumiar, Lote 06, Edifício Multitech - 2C1 Email: [email protected] 1600-546 Lisboa URL: http://www.dri.pt http://www.linux-geex.com Telefone: +351 21 715 30 55 Fax: +351 21 715 30 57 ---------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------ What You Don't Know About Data Connectivity CAN Hurt You This paper provides an overview of data connectivity, details its effect on application quality, and explores various alternative solutions. http://p.sf.net/sfu/progress-d2d _______________________________________________ BackupPC-users mailing list [email protected] List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
