Re: SSD caching an existing btrfs raid1
On 20/09/17 22:45, Kai Krakow wrote: Am Wed, 20 Sep 2017 17:51:15 +0200 schrieb Psalle: On 19/09/17 17:47, Austin S. Hemmelgarn wrote: (...) A better option if you can afford to remove a single device from that array temporarily is to use bcache. Bcache has one specific advantage in this case, multiple backend devices can share the same cache device. This means you don't have to carve out dedicated cache space for each disk on the SSD and leave some unused space so that you can add new devices if needed. The downside is that you can't convert each device in-place, but because you're using BTRFS, you can still convert the volume as a whole in-place. The procedure for doing so looks like this: 1. Format the SSD as a bcache cache. 2. Use `btrfs device delete` to remove a single hard drive from the array. 3. Set up the drive you just removed as a bcache backing device bound to the cache you created in step 1. 4. Add the new bcache device to the array. 5. Repeat from step 2 until the whole array is converted. A similar procedure can actually be used to do almost any underlying storage conversion (for example, switching to whole disk encryption, or adding LVM underneath BTRFS) provided all your data can fit on one less disk than you have. Thanks Austin, that's just great. For some reason I had discarded bcache thinking that it would force me to rebuild from scratch, but this kind of incremental migration is exactly why I hoped was possible. I have plenty of space to replace the devices one by one. I will report back my experience in a few days, I hope. I've done it exactly that way in the past and it worked flawlessly (but it took 24+ hours). But it was easy for me because I was also adding a third disk to the pool, so existing stuff could easily move. I suggest to initialize bcache to writearound mode while converting, so your maybe terabytes of disk don't go through the SSD. If you later decide to remove bcache or not sure about future bcache usage, you can wrap any partition into a bcache container - just don't connect it to a cache and it will work like a normal partition. Those are good advices. I've finished now and it seems to have gone without a hitch. Thanks! -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: SSD caching an existing btrfs raid1
> -Original Message- > From: linux-btrfs-ow...@vger.kernel.org [mailto:linux-btrfs- > ow...@vger.kernel.org] On Behalf Of Kai Krakow > Sent: Thursday, 21 September 2017 6:45 AM > To: linux-btrfs@vger.kernel.org > Subject: Re: SSD caching an existing btrfs raid1 > > Am Wed, 20 Sep 2017 17:51:15 +0200 > schrieb Psalle <psalleets...@gmail.com>: > > > On 19/09/17 17:47, Austin S. Hemmelgarn wrote: > > (...) > > > > > > A better option if you can afford to remove a single device from > > > that array temporarily is to use bcache. Bcache has one specific > > > advantage in this case, multiple backend devices can share the same > > > cache device. This means you don't have to carve out dedicated cache > > > space for each disk on the SSD and leave some unused space so that > > > you can add new devices if needed. The downside is that you can't > > > convert each device in-place, but because you're using BTRFS, you > > > can still convert the volume as a whole in-place. The procedure for > > > doing so looks like this: > > > > > > 1. Format the SSD as a bcache cache. > > > 2. Use `btrfs device delete` to remove a single hard drive from the > > > array. > > > 3. Set up the drive you just removed as a bcache backing device > > > bound to the cache you created in step 1. > > > 4. Add the new bcache device to the array. > > > 5. Repeat from step 2 until the whole array is converted. > > > > > > A similar procedure can actually be used to do almost any underlying > > > storage conversion (for example, switching to whole disk encryption, > > > or adding LVM underneath BTRFS) provided all your data can fit on > > > one less disk than you have. > > > > Thanks Austin, that's just great. For some reason I had discarded > > bcache thinking that it would force me to rebuild from scratch, but > > this kind of incremental migration is exactly why I hoped was > > possible. I have plenty of space to replace the devices one by one. > > > > I will report back my experience in a few days, I hope. > > I've done it exactly that way in the past and it worked flawlessly (but it > took > 24+ hours). But it was easy for me because I was also adding a third disk to > the pool, so existing stuff could easily move. Device delete takes freaking ages! I would avoid using it if you can. Device replace is much faster. Paul. N�r��yb�X��ǧv�^�){.n�+{�n�߲)w*jg����ݢj/���z�ޖ��2�ޙ&�)ߡ�a�����G���h��j:+v���w��٥
Re: SSD caching an existing btrfs raid1
Am Wed, 20 Sep 2017 17:51:15 +0200 schrieb Psalle: > On 19/09/17 17:47, Austin S. Hemmelgarn wrote: > (...) > > > > A better option if you can afford to remove a single device from > > that array temporarily is to use bcache. Bcache has one specific > > advantage in this case, multiple backend devices can share the same > > cache device. This means you don't have to carve out dedicated > > cache space for each disk on the SSD and leave some unused space so > > that you can add new devices if needed. The downside is that you > > can't convert each device in-place, but because you're using BTRFS, > > you can still convert the volume as a whole in-place. The > > procedure for doing so looks like this: > > > > 1. Format the SSD as a bcache cache. > > 2. Use `btrfs device delete` to remove a single hard drive from the > > array. > > 3. Set up the drive you just removed as a bcache backing device > > bound to the cache you created in step 1. > > 4. Add the new bcache device to the array. > > 5. Repeat from step 2 until the whole array is converted. > > > > A similar procedure can actually be used to do almost any > > underlying storage conversion (for example, switching to whole disk > > encryption, or adding LVM underneath BTRFS) provided all your data > > can fit on one less disk than you have. > > Thanks Austin, that's just great. For some reason I had discarded > bcache thinking that it would force me to rebuild from scratch, but > this kind of incremental migration is exactly why I hoped was > possible. I have plenty of space to replace the devices one by one. > > I will report back my experience in a few days, I hope. I've done it exactly that way in the past and it worked flawlessly (but it took 24+ hours). But it was easy for me because I was also adding a third disk to the pool, so existing stuff could easily move. I suggest to initialize bcache to writearound mode while converting, so your maybe terabytes of disk don't go through the SSD. If you later decide to remove bcache or not sure about future bcache usage, you can wrap any partition into a bcache container - just don't connect it to a cache and it will work like a normal partition. -- Regards, Kai Replies to list-only preferred. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: SSD caching an existing btrfs raid1
On 19/09/17 17:47, Austin S. Hemmelgarn wrote: (...) A better option if you can afford to remove a single device from that array temporarily is to use bcache. Bcache has one specific advantage in this case, multiple backend devices can share the same cache device. This means you don't have to carve out dedicated cache space for each disk on the SSD and leave some unused space so that you can add new devices if needed. The downside is that you can't convert each device in-place, but because you're using BTRFS, you can still convert the volume as a whole in-place. The procedure for doing so looks like this: 1. Format the SSD as a bcache cache. 2. Use `btrfs device delete` to remove a single hard drive from the array. 3. Set up the drive you just removed as a bcache backing device bound to the cache you created in step 1. 4. Add the new bcache device to the array. 5. Repeat from step 2 until the whole array is converted. A similar procedure can actually be used to do almost any underlying storage conversion (for example, switching to whole disk encryption, or adding LVM underneath BTRFS) provided all your data can fit on one less disk than you have. Thanks Austin, that's just great. For some reason I had discarded bcache thinking that it would force me to rebuild from scratch, but this kind of incremental migration is exactly why I hoped was possible. I have plenty of space to replace the devices one by one. I will report back my experience in a few days, I hope. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: SSD caching an existing btrfs raid1
Austin S. Hemmelgarn posted on Tue, 19 Sep 2017 11:47:24 -0400 as excerpted: > A better option if you can afford to remove a single device from that > array temporarily is to use bcache. Bcache has one specific advantage > in this case, multiple backend devices can share the same cache device. > This means you don't have to carve out dedicated cache space for each > disk on the SSD and leave some unused space so that you can add new > devices if needed. The downside is that you can't convert each device > in-place, but because you're using BTRFS, you can still convert the > volume as a whole in-place. The procedure for doing so looks like this: > > 1. Format the SSD as a bcache cache. > 2. Use `btrfs device delete` to remove a single hard drive from the > array. > 3. Set up the drive you just removed as a bcache backing device bound to > the cache you created in step 1. > 4. Add the new bcache device to the array. > 5. Repeat from step 2 until the whole array is converted. > > A similar procedure can actually be used to do almost any underlying > storage conversion (for example, switching to whole disk encryption, or > adding LVM underneath BTRFS) provided all your data can fit on one less > disk than you have. ... And you're not already at the minimum operational for that btrfs array type. For instance, I have lots of btrfs raid1s, two devices each. I'd have trouble with the above, because I can't simply remove a device, despite all the data and metadata fitting on a single device. I'd have three options that /would/ work, however. 1) If I was willing to risk having only a single checksummed copy of everything for the time it would take to process, I could do a btrfs balance convert from raid1 to single mode, then do a btrfs device remove, and go from there, converting back to raid1 after I was done. Of course if anything goes wrong with that single copy and something fails checksum, I've potentially lost whatever failed the checksum unless I have it on backup, so converting to single mode is risky. 2) If the data and metadata will fit on /half/ of a single device, then I could convert to dup mode (since btrfs now allows dup data chunks) taking double the space on that single device, but keeping two checksummed copies of everything due to the dup mode, thus avoiding the risk of #1. 3) If #2 won't work due to size and I was unwilling to risk #1, I'd have to add a device (making it the first one converted to bcache before the add) before I could remove one of the others, in ordered to allow me to keep the checksummed redundancy of raid1 for the entire procedure. After the bcache conversion and addition of the new device, and the removal, conversion, and readdition of one of the two existing devices, I could simply remove the third, giving me back my spare device, but I'd have to have and use it for the process. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: SSD caching an existing btrfs raid1
> -Original Message- > From: linux-btrfs-ow...@vger.kernel.org [mailto:linux-btrfs- > ow...@vger.kernel.org] On Behalf Of Pat Sailor > Sent: Wednesday, 20 September 2017 1:31 AM > To: Btrfs BTRFS <linux-btrfs@vger.kernel.org> > Subject: SSD caching an existing btrfs raid1 > > Hello, > > I have a half-filled raid1 on top of six spinning devices. Now I have come > into > a spare SSD I'd like to use for caching, if possible without having to > rebuild or, > failing that, without having to renounce to btrfs and flexible reshaping. > > I've been reading about the several options out there; I thought that > EnhanceIO would be the simplest bet but unfortunately I couldn't get it to > build with my recent kernel (last commits are from years ago). > > Failing that, I read that lvmcache could be the way to go. However, I can't > think of a way of setting it up in which I retain the ability to > add/remove/replace drives as I can do now with pure btrfs; if I opted to drop > btrfs to go to ext4 I still would have to offline the filesystem for > downsizes. > Not a frequent occurrence I hope, but now I'm used to keep working while I > reshape things in btrfs, and it's better if I can avoid large downtimes. > > Is what I want doable at all? Thanks in advance for any > suggestions/experiences to proceed. When I did mine I used a spare disk to create the initial LVM filesystem, then used btrfs dev replace to swap one of the raid1 mirrors to the new lvm device. Then repeat for the other mirror. My suggestion for lvm setup is to use two different pools, one for each btrfs mirror. That ensures you don't accidently have btrfs sharing the one physical disk by mistake, or lvm using the same SSD to cache the two discs. Paul.
Re: SSD caching an existing btrfs raid1
On 2017-09-19 11:30, Pat Sailor wrote: Hello, I have a half-filled raid1 on top of six spinning devices. Now I have come into a spare SSD I'd like to use for caching, if possible without having to rebuild or, failing that, without having to renounce to btrfs and flexible reshaping. I've been reading about the several options out there; I thought that EnhanceIO would be the simplest bet but unfortunately I couldn't get it to build with my recent kernel (last commits are from years ago). As a general rule, avoid things that are out of tree if you want any chance of support here. In the case of a third party module like EnhanceIO, you'll likely get asked to retest with a kernel without that module loaded if you run into a bug (because the module itself is suspect). Failing that, I read that lvmcache could be the way to go. However, I can't think of a way of setting it up in which I retain the ability to add/remove/replace drives as I can do now with pure btrfs; if I opted to drop btrfs to go to ext4 I still would have to offline the filesystem for downsizes. Not a frequent occurrence I hope, but now I'm used to keep working while I reshape things in btrfs, and it's better if I can avoid large downtimes. Is what I want doable at all? Thanks in advance for any suggestions/experiences to proceed. dm-cache (or lvmcache as the LVM2 developers want you to call it, despite it not being an LVM specific thing) would work fine, it won't prevent you from adding and removing devices, it would just make it more complicated. Without it, you just need to issue a replace command (or add then remove). With it, you need to set up the new cache device, bind that to the target, add the new device, and then delete the old device and remove the old cache target. dm-cache managed through LVM also has the advantage that you can convert the existing FS trivially, although you will have to take it off-line for the conversion. A better option if you can afford to remove a single device from that array temporarily is to use bcache. Bcache has one specific advantage in this case, multiple backend devices can share the same cache device. This means you don't have to carve out dedicated cache space for each disk on the SSD and leave some unused space so that you can add new devices if needed. The downside is that you can't convert each device in-place, but because you're using BTRFS, you can still convert the volume as a whole in-place. The procedure for doing so looks like this: 1. Format the SSD as a bcache cache. 2. Use `btrfs device delete` to remove a single hard drive from the array. 3. Set up the drive you just removed as a bcache backing device bound to the cache you created in step 1. 4. Add the new bcache device to the array. 5. Repeat from step 2 until the whole array is converted. A similar procedure can actually be used to do almost any underlying storage conversion (for example, switching to whole disk encryption, or adding LVM underneath BTRFS) provided all your data can fit on one less disk than you have. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
SSD caching an existing btrfs raid1
Hello, I have a half-filled raid1 on top of six spinning devices. Now I have come into a spare SSD I'd like to use for caching, if possible without having to rebuild or, failing that, without having to renounce to btrfs and flexible reshaping. I've been reading about the several options out there; I thought that EnhanceIO would be the simplest bet but unfortunately I couldn't get it to build with my recent kernel (last commits are from years ago). Failing that, I read that lvmcache could be the way to go. However, I can't think of a way of setting it up in which I retain the ability to add/remove/replace drives as I can do now with pure btrfs; if I opted to drop btrfs to go to ext4 I still would have to offline the filesystem for downsizes. Not a frequent occurrence I hope, but now I'm used to keep working while I reshape things in btrfs, and it's better if I can avoid large downtimes. Is what I want doable at all? Thanks in advance for any suggestions/experiences to proceed. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html