Re: [CentOS] Backup of large filesystems >500GB
On 5/27/2013 11:13 AM, Mike Watson wrote: > One > filesystem is very large, >500GB, and contains numerous large files: > SQL, docs, church libraries in ebook and digital form, plus stored > videos of church services. note that SQL database files generally can't be backed up safely while the SQL database server is active. either take a 'dump' or whatever backup of the database, and backup that dump rather than the raw SQL files, or stop the SQL service before making the backup, then restart it afterwards. Details vary per SQL server, of course. what would work really well for your general backup requirements, ignoring the above issue, is BackupPC. build an onsite backupPC server with a file system sufficiently large to hold all backups you want kept online (I generally do 2 weeks of incrementals), and setup BackupPC archiving to move copies of completed backups to an offsite repository for disaster recovery. -- john r pierce 37N 122W somewhere on the middle of the left coast ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup of large filesystems >500GB
On 05/27/2013 11:06 PM, Mark LaPierre wrote: > On 05/27/2013 02:13 PM, Mike Watson wrote: >> I've a small system that I use to support a number of churches. I >> provide web and email for them. My current server is running CentOS 6.3 >> with paired 1TB drives in a RAID1 configuration. It works well. One >> filesystem is very large, >500GB, and contains numerous large files: >> SQL, docs, church libraries in ebook and digital form, plus stored >> videos of church services. >> >> My problem is that I've found no means of backing up that file system. >> Dump and tar both error out as exceeding the maximum size. Neither will >> backup just the video directory (the largest) even with compression. >> >> Backup will be to an external (USB) removable HD. >> >> Can any suggest a prog or a method to back up this filesystem? >> >> mw >> > > I use rsync to make a daily back up of my data on a second drive that is > not normally mounted except when the backup is running. The drive is > inside the same box so this backup is still subject to loss if the box > is stolen or destroyed. I really should be backing up to an off site > location. > > If you have a fire, flood, or other general disaster your local backup > on an external drive isn't going to buy you anything unless you store > the external drive off site. > You can place that HDD in a rack (eSATA?) and add another that is kept off-site, and replace/swap them every week? Full off-site backup would require another box off-site with internet/wireless link for backups. Since you use rsync, data transfer should be fairly minimal. -- Ljubomir Ljubojevic (Love is in the Air) PL Computers Serbia, Europe StarOS, Mikrotik and CentOS/RHEL/Linux consultant ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup of large filesystems >500GB
How well does it run under cron? mw -- "Lose not thy airspeed, lest the ground rises up and smites thee." -- William Kershner http://crucis-court.com http://www.crucis.net/1632search On 05/28/2013 11:54 AM, Diego Sanchez wrote: > Or, you can use http://sourceforge.net/projects/ext2read/ (or similar) > > Copy&Paste: > > Ext2Read is an explorer like utility to explore ext2/ext3/ext4 files. It > now supports LVM2 and EXT4 extents. It can be used to view and copy files > and folders. It can recursively copy entire folders. It can also be used to > view and copy disk and file > > ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup of large filesystems >500GB
Or, you can use http://sourceforge.net/projects/ext2read/ (or similar) Copy&Paste: Ext2Read is an explorer like utility to explore ext2/ext3/ext4 files. It now supports LVM2 and EXT4 extents. It can be used to view and copy files and folders. It can recursively copy entire folders. It can also be used to view and copy disk and file -- Diego - Yo no soy paranoico! (pero que me siguen, me siguen) ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup of large filesystems >500GB
On Mon, May 27, 2013 at 9:41 PM, Les Mikesell wrote: > On Mon, May 27, 2013 at 7:05 PM, Mike Watson wrote: > > That's my problem. I hitting a file size limit with dump and tar. > > Reformatting is the obvious solution so you can use one of the > rsync-based backups - but if you really had to keep the FAT format you > could use tar |split -b with some reasonable size for output. > +1 for reformatting and using a file system that supports large(r) files Although not recommended... ... you could create a file with dd (that will take some time) on the NTFS drive, and then loopback mount it and format it ext3 (or whatever you choose). Then mount that and go on your merry way with your rsync backups. *Disclaimer*: This solution is ugly and I'm almost certainly I will get tomatoes thrown at me for suggesting it. ;) Which includes converting FAT32 to NTFS if you don't already have NTFS on the drive. There are more gotchas to what I suggested than just reformatting the drive ext3, etc and then sharing it out via Samba (or NFS) to your Windows clients (if you need to). > > -- >Les Mikesell > lesmikes...@gmail.com > ___ > CentOS mailing list > CentOS@centos.org > http://lists.centos.org/mailman/listinfo/centos > -- ---~~.~~--- Mike // SilverTip257 // ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup of large filesystems >500GB
On Mon, May 27, 2013 at 7:05 PM, Mike Watson wrote: > That's my problem. I hitting a file size limit with dump and tar. Reformatting is the obvious solution so you can use one of the rsync-based backups - but if you really had to keep the FAT format you could use tar |split -b with some reasonable size for output. -- Les Mikesell lesmikes...@gmail.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup of large filesystems >500GB
On 2013-05-27, Mike Watson wrote: > I'll check again, maybe NTSF. It's a singlepartition 1TB HD so it can't > be FAT32. Unless you have a desperate need for this disk to be readable by Windows, I would definitely make an ext3, ext4, or xfs filesystem on that USB drive. It is absolutely not worth the headache you'll have trying to restore properly from NTFS or FAT??. --keith -- kkel...@wombat.san-francisco.ca.us ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup of large filesystems >500GB
That's my problem. I hitting a file size limit with dump and tar. mw -- "Lose not thy airspeed, lest the ground rises up and smites thee." -- William Kershner http://crucis-court.com http://www.crucis.net/1632search On 05/27/2013 06:51 PM, Les Mikesell wrote: > On Mon, May 27, 2013 at 4:05 PM, Mike Watson wrote: >> I'll check again, maybe NTSF. It's a singlepartition 1TB HD so it can't >> be FAT32. >> > FAT32 can go to 2TB (you just can't format one that size in windows), > but has the 4GB file size limit. > > -- >Les Mikesell > lesmikes...@gmail.com > ___ > CentOS mailing list > CentOS@centos.org > http://lists.centos.org/mailman/listinfo/centos > ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup of large filesystems >500GB
On Mon, May 27, 2013 at 4:05 PM, Mike Watson wrote: > I'll check again, maybe NTSF. It's a singlepartition 1TB HD so it can't > be FAT32. > FAT32 can go to 2TB (you just can't format one that size in windows), but has the 4GB file size limit. -- Les Mikesell lesmikes...@gmail.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup of large filesystems >500GB
On 05/27/2013 02:13 PM, Mike Watson wrote: > I've a small system that I use to support a number of churches. I > provide web and email for them. My current server is running CentOS 6.3 > with paired 1TB drives in a RAID1 configuration. It works well. One > filesystem is very large, >500GB, and contains numerous large files: > SQL, docs, church libraries in ebook and digital form, plus stored > videos of church services. > > My problem is that I've found no means of backing up that file system. > Dump and tar both error out as exceeding the maximum size. Neither will > backup just the video directory (the largest) even with compression. > > Backup will be to an external (USB) removable HD. > > Can any suggest a prog or a method to back up this filesystem? > > mw > I use rsync to make a daily back up of my data on a second drive that is not normally mounted except when the backup is running. The drive is inside the same box so this backup is still subject to loss if the box is stolen or destroyed. I really should be backing up to an off site location. If you have a fire, flood, or other general disaster your local backup on an external drive isn't going to buy you anything unless you store the external drive off site. -- _ °v° /(_)\ ^ ^ Mark LaPierre Registered Linux user No #267004 https://linuxcounter.net/ ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup of large filesystems >500GB
I'll check again, maybe NTSF. It's a singlepartition 1TB HD so it can't be FAT32. mw -- "Lose not thy airspeed, lest the ground rises up and smites thee." -- William Kershner http://crucis-court.com http://www.crucis.net/1632search On 05/27/2013 01:43 PM, Robert Nichols wrote: > On 05/27/2013 01:13 PM, Mike Watson wrote: >> Backup will be to an external (USB) removable HD. > What file system is on that external HD? FAT32 has a 4GB limit > for file size. > ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup of large filesystems >500GB
I've used rsync for remote file xfer of directory trees. It's been awhile, I'd forgotten about it. mw -- "Lose not thy airspeed, lest the ground rises up and smites thee." -- William Kershner http://crucis-court.com http://www.crucis.net/1632search On 05/27/2013 01:24 PM, Nux! wrote: > On 27.05.2013 19:13, Mike Watson wrote: >> I've a small system that I use to support a number of churches. I >> provide web and email for them. My current server is running CentOS >> 6.3 >> with paired 1TB drives in a RAID1 configuration. It works well. One >> filesystem is very large, >500GB, and contains numerous large files: >> SQL, docs, church libraries in ebook and digital form, plus stored >> videos of church services. >> >> My problem is that I've found no means of backing up that file system. >> Dump and tar both error out as exceeding the maximum size. Neither >> will >> backup just the video directory (the largest) even with compression. >> >> Backup will be to an external (USB) removable HD. >> >> Can any suggest a prog or a method to back up this filesystem? > Have you tried good old rsync? Or, if you want incremental backups, > check rdiff-backup. > I'm sure our list colleagues will come up with even more solutions. > ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup of large filesystems >500GB
On Mon, May 27, 2013 at 2:55 PM, Keith Keller < kkel...@wombat.san-francisco.ca.us> wrote: > On 2013-05-27, Mike Watson wrote: > > I've a small system that I use to support a number of churches. I > > provide web and email for them. My current server is running CentOS 6.3 > > with paired 1TB drives in a RAID1 configuration. It works well. One > > filesystem is very large, >500GB, and contains numerous large files: > > SQL, docs, church libraries in ebook and digital form, plus stored > > videos of church services. > > > > Backup will be to an external (USB) removable HD. > > > > Can any suggest a prog or a method to back up this filesystem? > > People have already suggested rsync and rdiff-backup; there's also > rsnapshot which is built on top of rsync. > > Another option could be mdadm if your RAID1 is already an mdadm array. > You can add your USB drive to the array, wait for it to rebuild, then > remove it from the array. I'd be wary of backing up an SQL database in > that way, but I'd be wary of using rsync, dump, or tar too, so be sure > to take a real backup (e.g., mysqldump, pg_dump) of your database first. > > --keith > > -- > kkel...@wombat.san-francisco.ca.us > > > ___ > CentOS mailing list > CentOS@centos.org > http://lists.centos.org/mailman/listinfo/centos > I'd take the extra step of reformatting the USB drive into an ext3 filesystem, then just use rsync in a nightly cron job. We do it with about a dozen or so workstations here with user data either on internal disks, or other external USB drives. Works great. -- Matt Phelps System Administrator, Computation Facility Harvard - Smithsonian Center for Astrophysics mphe...@cfa.harvard.edu, http://www.cfa.harvard.edu ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup of large filesystems >500GB
On 2013-05-27, Mike Watson wrote: > I've a small system that I use to support a number of churches. I > provide web and email for them. My current server is running CentOS 6.3 > with paired 1TB drives in a RAID1 configuration. It works well. One > filesystem is very large, >500GB, and contains numerous large files: > SQL, docs, church libraries in ebook and digital form, plus stored > videos of church services. > > Backup will be to an external (USB) removable HD. > > Can any suggest a prog or a method to back up this filesystem? People have already suggested rsync and rdiff-backup; there's also rsnapshot which is built on top of rsync. Another option could be mdadm if your RAID1 is already an mdadm array. You can add your USB drive to the array, wait for it to rebuild, then remove it from the array. I'd be wary of backing up an SQL database in that way, but I'd be wary of using rsync, dump, or tar too, so be sure to take a real backup (e.g., mysqldump, pg_dump) of your database first. --keith -- kkel...@wombat.san-francisco.ca.us ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup of large filesystems >500GB
On 05/27/2013 01:13 PM, Mike Watson wrote: > Backup will be to an external (USB) removable HD. What file system is on that external HD? FAT32 has a 4GB limit for file size. -- Bob Nichols "NOSPAM" is really part of my email address. Do NOT delete it. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup of large filesystems >500GB
On 27.05.2013 19:13, Mike Watson wrote: > I've a small system that I use to support a number of churches. I > provide web and email for them. My current server is running CentOS > 6.3 > with paired 1TB drives in a RAID1 configuration. It works well. One > filesystem is very large, >500GB, and contains numerous large files: > SQL, docs, church libraries in ebook and digital form, plus stored > videos of church services. > > My problem is that I've found no means of backing up that file system. > Dump and tar both error out as exceeding the maximum size. Neither > will > backup just the video directory (the largest) even with compression. > > Backup will be to an external (USB) removable HD. > > Can any suggest a prog or a method to back up this filesystem? Have you tried good old rsync? Or, if you want incremental backups, check rdiff-backup. I'm sure our list colleagues will come up with even more solutions. -- Sent from the Delta quadrant using Borg technology! Nux! www.nux.ro ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup of large filesystems >500GB
On Mon, 27 May 2013 13:13:30 -0500 Mike Watson wrote: > Can any suggest a prog or a method to back up this filesystem? rsync -- MELVILLE THEATRE ~ Real D 3D Digital Cinema ~ www.melvilletheatre.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
