Re: rsync backup to ext3-formatted usb flash drive?
On Aug 10, 2008, at 2:12 AM, Brian Wells wrote: Hi. I'm using svn-fast-backup (in the subversion-tools package) to make rsync backups of my subversion repository. The place I'm backing up to is on an ext3-formatted usb flash drive, and I'm wondering if this is a wise thing to do. I know that flash drive blocks wear out after a certain number of writes, and rsync uses hard links to avoid duplicate backup files. My question is, does this mean that the inodes' reference counts will be changed frequently enough that it might wear out that block of the flash drive any time soon? (I currently backup about 900 files one or two times a day.) If so, is ext3 capable of using another block for the inode, or will I lose the ability to read/write some files completely? Thanks, Brian Here are some basic facts about USB flash drives: The underlying NAND flash RAM is guaranteed for 100K writes (older technology) or 1M (more recent). After that it's retention time starts to decrease, sometimes to as little as a few weeks, and with continued use eventually to complete failure. The retention time of a new, fully-functional NAND flash RAM is hundreds of years. USB flash drives have controller chips in the drive that, among other things, talk USB protocol to the host. One of the other things they do is try to distribute the writes over the available blocks, extending the lifetime of the device beyond the raw lifetime of the underlying NAND flash. But the firmware in the controller usually assumes you are using a FAT file-system. If, for example, you use EXT3 rather than FAT, you will probably confuse it. I don't know what the effect will be on device life, but I'm guessing it will probably be something like a return to the behavior of the raw NAND flash. Write blocks are large (4-8K or more). If you update one inode in a block, you have to read the entire block, update the inode and write the entire block back again. Each write to an inode causes an effective write to all the inodes in the same block. So there's roughly a 64x multiplier (8Kbytes/block divided by 128bytes/inode = 64 inodes/block) in the number of writes. Thus any given single inode is good for roughly 16K updates before it wears out. Let's suppose your rsync backup causes a small number of updates for each inode it touches. Call it 10, conservatively (it very well could be as low as 1 or 2 -- I don't know enough of the details of how rsync works at the filesystem level). That gives you about 1600 rsync runs before you start to wear out the inodes. At twice a day, you can safely use your USB flash drive for 800 days, or between 2 and 3 years. So here's what I'd do. Use your USB flash drive for a year, then retire it to the long-term archives and replace it with a new one. Let that be one of those things you do on or near your birthday, like schedule a checkup with your Doctor. Also: dismount the flash drive, force an fsck, and remount it, every time you do a backup. This will read and sanity-check every inode on the drive, without doing any writes. If the fsck shows random errors, replace the drive because it's probably starting to wear out. Sound reasonable? Rick -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: rsync backup to ext3-formatted usb flash drive?
On Mon, 2008-08-11 at 02:21 -0400, Rick Thomas wrote: On Aug 10, 2008, at 2:12 AM, Brian Wells wrote: Hi. I'm using svn-fast-backup (in the subversion-tools package) to make rsync backups of my subversion repository. The place I'm backing up to is on an ext3-formatted usb flash drive, and I'm wondering if this is a wise thing to do. I know that flash drive blocks wear out after a certain number of writes, and rsync uses hard links to avoid duplicate backup files. My question is, does this mean that the inodes' reference counts will be changed frequently enough that it might wear out that block of the flash drive any time soon? (I currently backup about 900 files one or two times a day.) If so, is ext3 capable of using another block for the inode, or will I lose the ability to read/write some files completely? Thanks, Brian Here are some basic facts about USB flash drives: The underlying NAND flash RAM is guaranteed for 100K writes (older technology) or 1M (more recent). After that it's retention time starts to decrease, sometimes to as little as a few weeks, and with continued use eventually to complete failure. The retention time of a new, fully-functional NAND flash RAM is hundreds of years. USB flash drives have controller chips in the drive that, among other things, talk USB protocol to the host. One of the other things they do is try to distribute the writes over the available blocks, extending the lifetime of the device beyond the raw lifetime of the underlying NAND flash. But the firmware in the controller usually assumes you are using a FAT file-system. If, for example, you use EXT3 rather than FAT, you will probably confuse it. I don't know what the effect will be on device life, but I'm guessing it will probably be something like a return to the behavior of the raw NAND flash. Write blocks are large (4-8K or more). If you update one inode in a block, you have to read the entire block, update the inode and write the entire block back again. Each write to an inode causes an effective write to all the inodes in the same block. So there's roughly a 64x multiplier (8Kbytes/block divided by 128bytes/inode = 64 inodes/block) in the number of writes. Thus any given single inode is good for roughly 16K updates before it wears out. Let's suppose your rsync backup causes a small number of updates for each inode it touches. Call it 10, conservatively (it very well could be as low as 1 or 2 -- I don't know enough of the details of how rsync works at the filesystem level). That gives you about 1600 rsync runs before you start to wear out the inodes. At twice a day, you can safely use your USB flash drive for 800 days, or between 2 and 3 years. So here's what I'd do. Use your USB flash drive for a year, then retire it to the long-term archives and replace it with a new one. Let that be one of those things you do on or near your birthday, like schedule a checkup with your Doctor. Also: dismount the flash drive, force an fsck, and remount it, every time you do a backup. This will read and sanity-check every inode on the drive, without doing any writes. If the fsck shows random errors, replace the drive because it's probably starting to wear out. Sound reasonable? Rick It does sound reasonable, with one sticking point: a lot of space in the flash drive will never be used. I should have mentioned that; despite having 900 files, the repository is currently only about 5 MB, and the drive itself is 4 GB. I'm looking at an alternative utility, svn-backup-dumps, in the same package. I could do ~800 full backups, or many more incremental backups with a full backup here and there, before running out of space. That would only write most blocks once or twice, and once it filled up I could erase old backups with only one write and reuse it, right? (If I do this, should I reformat to FAT so the controllers won't be confused?) Thanks, Brian -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: rsync backup to ext3-formatted usb flash drive?
Brian Wells wrote: On Mon, 2008-08-11 at 02:21 -0400, Rick Thomas wrote: On Aug 10, 2008, at 2:12 AM, Brian Wells wrote: Hi. I'm using svn-fast-backup (in the subversion-tools package) to make rsync backups of my subversion repository. The place I'm backing up to is on an ext3-formatted usb flash drive, and I'm wondering if this is a wise thing to do. I know that flash drive blocks wear out after a certain number of writes, and rsync uses hard links to avoid duplicate backup files. My question is, does this mean that the inodes' reference counts will be changed frequently enough that it might wear out that block of the flash drive any time soon? (I currently backup about 900 files one or two times a day.) If so, is ext3 capable of using another block for the inode, or will I lose the ability to read/write some files completely? Thanks, Brian Here are some basic facts about USB flash drives: The underlying NAND flash RAM is guaranteed for 100K writes (older technology) or 1M (more recent). After that it's retention time starts to decrease, sometimes to as little as a few weeks, and with continued use eventually to complete failure. The retention time of a new, fully-functional NAND flash RAM is hundreds of years. USB flash drives have controller chips in the drive that, among other things, talk USB protocol to the host. One of the other things they do is try to distribute the writes over the available blocks, extending the lifetime of the device beyond the raw lifetime of the underlying NAND flash. But the firmware in the controller usually assumes you are using a FAT file-system. If, for example, you use EXT3 rather than FAT, you will probably confuse it. I don't know what the effect will be on device life, but I'm guessing it will probably be something like a return to the behavior of the raw NAND flash. Write blocks are large (4-8K or more). If you update one inode in a block, you have to read the entire block, update the inode and write the entire block back again. Each write to an inode causes an effective write to all the inodes in the same block. So there's roughly a 64x multiplier (8Kbytes/block divided by 128bytes/inode = 64 inodes/block) in the number of writes. Thus any given single inode is good for roughly 16K updates before it wears out. Let's suppose your rsync backup causes a small number of updates for each inode it touches. Call it 10, conservatively (it very well could be as low as 1 or 2 -- I don't know enough of the details of how rsync works at the filesystem level). That gives you about 1600 rsync runs before you start to wear out the inodes. At twice a day, you can safely use your USB flash drive for 800 days, or between 2 and 3 years. So here's what I'd do. Use your USB flash drive for a year, then retire it to the long-term archives and replace it with a new one. Let that be one of those things you do on or near your birthday, like schedule a checkup with your Doctor. Also: dismount the flash drive, force an fsck, and remount it, every time you do a backup. This will read and sanity-check every inode on the drive, without doing any writes. If the fsck shows random errors, replace the drive because it's probably starting to wear out. Sound reasonable? Rick It does sound reasonable, with one sticking point: a lot of space in the flash drive will never be used. I should have mentioned that; despite having 900 files, the repository is currently only about 5 MB, and the drive itself is 4 GB. I'm looking at an alternative utility, svn-backup-dumps, in the same package. I could do ~800 full backups, or many more incremental backups with a full backup here and there, before running out of space. That would only write most blocks once or twice, and once it filled up I could erase old backups with only one write and reuse it, right? (If I do this, should I reformat to FAT so the controllers won't be confused?) Thanks, Brian It seems to me (given the above remarks) that you could define a block or file that contains your backup with suitable space for the year (as already suggested). I think using the FAT for filename suggestion, at the end of each year make the block containing your backup a read only block. FAT does not have the permissions as does EXT3 but can be set for read only operation. You can easily try this out using a test block. Each year (since you were concerned about unused space) you would then define a new block with the beginning of the new block as the contents of the previous block unless of course you don't want to keep all the old backup info. Hope you find this suggestion helpful. Thanks, Ted Hilts -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: rsync backup to ext3-formatted usb flash drive?
On Monday 11 August 2008 23:25, Brian Wells wrote: I'm looking at an alternative utility, svn-backup-dumps, in the same package. I could do ~800 full backups, or many more incremental backups with a full backup here and there, before running out of space. That would only write most blocks once or twice, and once it filled up I could erase old backups with only one write and reuse it, right? (If I do this, should I reformat to FAT so the controllers won't be confused?) If you're worried about not confusing the controllers, you could create a vfat fs normally in the device and then make a file in it, like dd if=/dev/zero of=/media/usbdisk/ext3.fs bs=1M count=4000 and then make an ext3 filesystem in that file, like mkfs.ext3 -L my_backup_fs /media/usbdisk/ext3.fs You will be able to mount this file with the -o loop option and make your backups there :) But anyways... I'm not sure that those controllers even deal with data in fs-level. Thanks, Brian -- Shachar Or | שחר אור http://ox.freeallweb.org/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: rsync backup to ext3-formatted usb flash drive?
On Aug 11, 2008, at 4:25 PM, Brian Wells wrote: I'm looking at an alternative utility, svn-backup-dumps, in the same package. I could do ~800 full backups, or many more incremental backups with a full backup here and there, before running out of space. That would only write most blocks once or twice, and once it filled up I could erase old backups with only one write and reuse it, right? (If I do this, should I reformat to FAT so the controllers won't be confused?) The inodes are going to be your problem in an ext3 filesystem, not the data blocks. Keep that in mind. So you should be trying to control the number of writes per inode. One way to get a better handle on that is by using the noatime mount option. Or switching to FAT, if your backup system supports FAT as a destination filesystem. USB flash drives are not expensive -- cdw.com advertises a variety of 4GB drives for US$15-$20, and they aren't the cheapest by a good shot. No matter what you finally wind up doing, I'd still plan to replace the flash drive at least once a year. That works out to about 5 US cents/day. Rick -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: rsync backup to ext3-formatted usb flash drive?
Thanks to Shachar Or and Ted Hilts for the suggestions, I might try them sometime (particularly the nested filesystem idea, that's very clever :) My main reply, below, is to Rick. On Mon, 2008-08-11 at 18:33 -0400, Rick Thomas wrote: On Aug 11, 2008, at 4:25 PM, Brian Wells wrote: I'm looking at an alternative utility, svn-backup-dumps, in the same package. I could do ~800 full backups, or many more incremental backups with a full backup here and there, before running out of space. That would only write most blocks once or twice, and once it filled up I could erase old backups with only one write and reuse it, right? (If I do this, should I reformat to FAT so the controllers won't be confused?) The inodes are going to be your problem in an ext3 filesystem, not the data blocks. Keep that in mind. So you should be trying to control the number of writes per inode. One way to get a better handle on that is by using the noatime mount option. Or switching to FAT, if your backup system supports FAT as a destination filesystem. USB flash drives are not expensive -- cdw.com advertises a variety of 4GB drives for US$15-$20, and they aren't the cheapest by a good shot. No matter what you finally wind up doing, I'd still plan to replace the flash drive at least once a year. That works out to about 5 US cents/day. Rick Looks like you caught some wishful thinking on my part. I guessed that the inode blocks would still take a lot of writes, hence most blocks instead of all blocks; but I don't know enough details about whether inodes in ext3 are kept close together (i.e. in the same block) or not. If they are not, then adding new files should create inodes in another block, which shouldn't cause excessive writes to any single block. (The access time change might, though.) Again, I don't know enough to be sure, but I was hoping it might be so. Interestingly, while I was researching on the web I found another issue: the journal that ext3 keeps. Quoting http://en.wikipedia.org/wiki/Ext2 ---Begin Quote--- ext2 is still recommended over journaling file systems on bootable USB flash drives and other solid-state drives. ext2 performs fewer writes than ext3 since it does not need to write to the journal. As the major aging factor of a flash chip is the number of erase cycles, and as those happen frequently on writes, this increases the life span of the solid-state device. Another good practice for filesystems on flash devices is the use of the noatime mount option, for the same reason. ---End Quote--- So it looks like there's another factor that might run the drive into the ground. Wonderful, I think I see why most people keep FAT on them. On the noatime issue, GNOME in etch mounts the drive automatically for me when I plug it in; I'm not sure how to configure GNOME to use custom mount options, short of unmounting and remounting manually every time I stick the drive in. I think I'm also seeing one kind of reason that many people hate automagical GUIs. Can someone point me to the docs? Thanks, Brian -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
rsync backup to ext3-formatted usb flash drive?
Hi. I'm using svn-fast-backup (in the subversion-tools package) to make rsync backups of my subversion repository. The place I'm backing up to is on an ext3-formatted usb flash drive, and I'm wondering if this is a wise thing to do. I know that flash drive blocks wear out after a certain number of writes, and rsync uses hard links to avoid duplicate backup files. My question is, does this mean that the inodes' reference counts will be changed frequently enough that it might wear out that block of the flash drive any time soon? (I currently backup about 900 files one or two times a day.) If so, is ext3 capable of using another block for the inode, or will I lose the ability to read/write some files completely? Thanks, Brian -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
