Re: btrfs device delete /dev/sdc1 /mnt/raid1 user experience

[Chris Murphy]

On Mon, Jun 6, 2016 at 10:02 PM, Kai Hendry  wrote:
> Sorry I unsubscribed from linux-btrfs@vger.kernel.org since the traffic
> was a bit too high for me.
>
> On Tue, 7 Jun 2016, at 11:42 AM, Chris Murphy wrote:
>> Your command turned this from a 3 drive volume into a 2 drive volume,
>> it removed the drive you asked to be removed.
>
> I actually had 2 drives to begin with, but it wouldn't allow me to
> delete the drive without adding a new one, so I had 3 drives.

Two drive raid1 is the minimum number of drives, that's why it won't
allow you to delete one.


>
> I think I've confused you by my talk of "RAID-1 over 3 drives". I am not
> sure why RAIDX terminology is so confusing. All I want is for the data
> on one drive to be mirrored across them all. So if I add X drives, I
> want to X exact copies. But IIUC this changes the RAID level and
> everyone gets confused what I am talking about.

Yeah you're confused about both RAID in general and how Btrfs raid
works. There is no guarantee of X drives meaning X copies with any
RAID-1. The mkfs.btrfs man page makes it very clear that there are
only ever two copies of data with Btrfs raid1 no matter how many
drives you include.


>
>
>> You're definitely missing something but once you understand how it
>> actually works, it'll be interesting if you have suggestions on how
>> the existing documentation confused you into making this mistake.
>
>
> Summary: I issued the remove command when I should have just removed it
> physically IIUC.

Yes, but that only works with a two drive Btrfs raid1. The minimum
number of drives for successful degraded mount is n-1 drives. You can
only lose 1 drive with Btrfs raid1 volumes, you must have all of the
other drives for it to be mountable degraded.



> If I was to do this again, I would unmount the raid1 mount. Take a disk
> physically out and then add the new one. So yes, it would be degraded
> but then as you mention, the newly drive will sort it out. One would
> hope.

Btrfs isn't mature enough to hope for the best. You definitely need to
understand it. There's a lot less you need to understand than mdadm or
LVM plus other file system stuff, but there's still minimum knowledge
required and also there are gotchas that aren't always obvious.

-- 
Chris Murphy
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs device delete /dev/sdc1 /mnt/raid1 user experience

[Austin S. Hemmelgarn]

On 2016-06-07 09:52, Kai Hendry wrote:

On Tue, 7 Jun 2016, at 07:10 PM, Austin S. Hemmelgarn wrote:

Yes, although you would then need to be certain to run a balance with
-dconvert=raid1 -mconvert=raid1 to clean up anything that got allocated
before the new disk was added.


I don't quite understand when I should run this command.

Before I take it out? Once I take it out? One the new one is added?
On both systems once you've gotten the new devices added so that the 
filesystems are no longer degraded.


And why doesn't it do this automatically? =)
Ideally, it shouldn't be needed at all, it's just needed due to a 
deficiency in the high-level allocator in BTRFS.  Once that is fixed, 
this won't be needed at all, and I think the consensus is that that 
should be fixed instead of adding in something that will need to be 
taken out again later.


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs device delete /dev/sdc1 /mnt/raid1 user experience

[Kai Hendry]

On Tue, 7 Jun 2016, at 07:10 PM, Austin S. Hemmelgarn wrote:
> Yes, although you would then need to be certain to run a balance with 
> -dconvert=raid1 -mconvert=raid1 to clean up anything that got allocated 
> before the new disk was added.

I don't quite understand when I should run this command.

Before I take it out? Once I take it out? One the new one is added?

And why doesn't it do this automatically? =)
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs device delete /dev/sdc1 /mnt/raid1 user experience

[Austin S. Hemmelgarn]

On 2016-06-07 00:02, Kai Hendry wrote:

Sorry I unsubscribed from linux-btrfs@vger.kernel.org since the traffic
was a bit too high for me.
Entirely understandable, although for what it's worth it's nowhere near 
as busy as some other mailing lists (linux-ker...@vger.kernel.org for 
example sees well over a thousand messages a day on average).


On Tue, 7 Jun 2016, at 11:42 AM, Chris Murphy wrote:

Your command turned this from a 3 drive volume into a 2 drive volume,
it removed the drive you asked to be removed.


I actually had 2 drives to begin with, but it wouldn't allow me to
delete the drive without adding a new one, so I had 3 drives.

I think I've confused you by my talk of "RAID-1 over 3 drives". I am not
sure why RAIDX terminology is so confusing. All I want is for the data
on one drive to be mirrored across them all. So if I add X drives, I
want to X exact copies. But IIUC this changes the RAID level and
everyone gets confused what I am talking about.
Actually, because of how raid1 mode in BTRFS works, you just get 2 
copies no matter how many drives you have.  Once you add more than the 
first two drives, it starts to work more like RAID-10, although slightly 
less efficiently than the raid10 profile.  This is why you couldn't pull 
a single drive out of a 3 drive raid1 set and have all the data.




You're definitely missing something but once you understand how it
actually works, it'll be interesting if you have suggestions on how
the existing documentation confused you into making this mistake.



Summary: I issued the remove command when I should have just removed it
physically IIUC.
Yes.  The device delete command is for shrinking arrays, removing 
failing disks, or for re-purposing individual drives.




If I was to do this again, I would unmount the raid1 mount. Take a disk
physically out and then add the new one. So yes, it would be degraded
but then as you mention, the newly drive will sort it out. One would
hope.
Yes, although you would then need to be certain to run a balance with 
-dconvert=raid1 -mconvert=raid1 to clean up anything that got allocated 
before the new disk was added.


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs device delete /dev/sdc1 /mnt/raid1 user experience

[Kai Hendry]

Sorry I unsubscribed from linux-btrfs@vger.kernel.org since the traffic
was a bit too high for me.

On Tue, 7 Jun 2016, at 11:42 AM, Chris Murphy wrote:
> Your command turned this from a 3 drive volume into a 2 drive volume,
> it removed the drive you asked to be removed.

I actually had 2 drives to begin with, but it wouldn't allow me to
delete the drive without adding a new one, so I had 3 drives.

I think I've confused you by my talk of "RAID-1 over 3 drives". I am not
sure why RAIDX terminology is so confusing. All I want is for the data
on one drive to be mirrored across them all. So if I add X drives, I
want to X exact copies. But IIUC this changes the RAID level and
everyone gets confused what I am talking about.


> You're definitely missing something but once you understand how it
> actually works, it'll be interesting if you have suggestions on how
> the existing documentation confused you into making this mistake.


Summary: I issued the remove command when I should have just removed it
physically IIUC.



If I was to do this again, I would unmount the raid1 mount. Take a disk
physically out and then add the new one. So yes, it would be degraded
but then as you mention, the newly drive will sort it out. One would
hope.

Thanks,
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs device delete /dev/sdc1 /mnt/raid1 user experience

[Chris Murphy]

On Mon, Jun 6, 2016 at 7:19 PM, Kai Hendry  wrote:
> On Mon, 6 Jun 2016, at 10:16 PM, Austin S. Hemmelgarn wrote:
>> Based on the fact that you appear to want to carry a disk to copy data
>> more quickly than over then internet, then what you've already done plus
>> this is the correct way to do it.
>
> The trouble is the way I ended up doing it:
> 1) Replacing it with a bigger disk
> 2) Deleting the disk
> 3) btrfs send /mnt/raid1/snapshot | btrfs receive /mnt/onetb/
>
> This took almost **an entire day** over USB3 spinning disks  !
>
> My expectation was that I could quickly remove a mirrored disk, replace
> that disk and hand carry the disk I removed. btrfs shouldn't remove data
> from the disk I removed.

Had this been a two drive raid1, all you need to do is take one drive
with you. No commands necessary. Each drive can be separately mounted
with -o degraded, and you can re-establish two copy replication with
'btrfs dev add' with a new drive and 'btrfs dev del missing' to get
rid of the phantom missing drive. You'd do that locally and then also
at your destination.

A gotcha with this technique though is often single chunks get created
during the -o degraded read-write mount. After 'dev del missing'
completes the replication process, you need to check 'fi df' or 'fi
us' for single chunks. If they exist, get rid of them with a filtered
balance. '-dconvert=raid1,soft -mconvert=raid1,soft' should work I
think. If there are single chunks created and not converted, later if
you need to do a degraded mount it will fail. Usually it can be made
to work with -o ro,degraded in such a case, but it means no more
read-write for the file system, you'd have to recreate it (at this
point until it's fixed).


-- 
Chris Murphy
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs device delete /dev/sdc1 /mnt/raid1 user experience

[Chris Murphy]

On Sun, Jun 5, 2016 at 11:44 PM, Kai Hendry  wrote:
> Hi there,
>
>
> I planned to remove one of my disks, so that I can take it from
> Singapore to the UK and then re-establish another remote RAID1 store.
>
> delete is an alias of remove, so I added a new disk (devid 3) and
> proceeded to run:
> btrfs device delete /dev/sdc1 /mnt/raid1 (devid 1)

Wrong command. This means you don't want sdc1 used in this volume
anymore and in the process it is invalidated because to remove it from
Btrfs means to remove its signature.





>
>
> nuc:~$ uname -a
> Linux nuc 4.5.4-1-ARCH #1 SMP PREEMPT Wed May 11 22:21:28 CEST 2016
> x86_64 GNU/Linux
> nuc:~$   btrfs --version
> btrfs-progs v4.5.3
> nuc:~$ sudo btrfs fi show /mnt/raid1/
> Label: 'extraid1'  uuid: 5cab2a4a-e282-4931-b178-bec4c73cdf77
> Total devices 2 FS bytes used 776.56GiB
> devid2 size 931.48GiB used 778.03GiB path /dev/sdb1
> devid3 size 1.82TiB used 778.03GiB path /dev/sdd1

OK I'm confused. You had a three drive Btrfs raid-1 and you expected
to take 1 drive on a trip to establish that data elsewhere? That's not
possible. The minimum number of drives for degraded mount of a 3 drive
Btrfs raid1 is 2 drives. There aren't three copies of the data with a
three drive raid1. There are two copies only spread across three
drives.

Your command turned this from a 3 drive volume into a 2 drive volume,
it removed the drive you asked to be removed.



> First off, I was expecting btrfs to release the drive pretty much
> immediately. The command took about half a day to complete. I watched
> `btrfs fi show` to see size of devid 1 (the one I am trying to remove)
> to be zero and to see used space slowly go down whilst used space of
> devid 3 (the new disk) slowly go up.

Expected behavior, it was told you no longer wanted a 3 drive raid1,
so the data on the removed drive was being replicated onto the other
two drives to maintain two copies of your data on those two drives.


> Secondly and most importantly my /dev/sdc1 can't be mounted now anymore.
> Why?

The data that was on that drive is still there, just the magic was
invalidated as the final step in the operation. But that one drive
doesn't contain all of your data anyway so by itself the one drive
won't mount degraded, too many devices are missing.


>
> sudo mount -t btrfs /dev/sdc1 /mnt/test/
> mount: wrong fs type, bad option, bad superblock on /dev/sdc1,
>missing codepage or helper program, or other error
>
>In some cases useful info is found in syslog - try
>dmesg | tail or so.
>
> There is nothing in dmesg nor my journal. I wasn't expecting my drive to
> be rendered useless on removing or am I missing something?

You're definitely missing something but once you understand how it
actually works, it'll be interesting if you have suggestions on how
the existing documentation confused you into making this mistake.


> I'm still keen to take a TB on a flight with me the day after tomorrow.
> What is the recommended course of action? Recreate a mkfs.btrfs on
> /dev/sdc1 and send data to it from /mnt/raid1?

You can do that. The 2 disk raid1 can be used as the send, and the new
fs on the removed 3rd drive can be used as the receive side. That
would be per subvolume however.

It would be faster to use a btrfs seed device for this. But as I went
through the steps, I got confused how to remove the two raid1s. Both
are read only in this case, so I don't know what happens, it'd require
testing and the seed > sprout stuff isn't as well tested. But at least
it's relatively safe since the original is flagged read only.


> Still I hope the experience could be improved to remove a disk sanely.

Yeah but we'll have to understand at what point you're confused. It
seems like maybe you're not aware that  three drive raid1 still means
two copies of data, not three. Taking one drive would never have
worked anyway.

-- 
Chris Murphy
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs device delete /dev/sdc1 /mnt/raid1 user experience

[Kai Hendry]

On Mon, 6 Jun 2016, at 10:16 PM, Austin S. Hemmelgarn wrote:
> Based on the fact that you appear to want to carry a disk to copy data 
> more quickly than over then internet, then what you've already done plus 
> this is the correct way to do it.

The trouble is the way I ended up doing it:
1) Replacing it with a bigger disk
2) Deleting the disk
3) btrfs send /mnt/raid1/snapshot | btrfs receive /mnt/onetb/

This took almost **an entire day** over USB3 spinning disks  !

My expectation was that I could quickly remove a mirrored disk, replace
that disk and hand carry the disk I removed. btrfs shouldn't remove data
from the disk I removed.

Kind regards,
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs device delete /dev/sdc1 /mnt/raid1 user experience

[Austin S. Hemmelgarn]

On 2016-06-06 01:44, Kai Hendry wrote:

Hi there,


I planned to remove one of my disks, so that I can take it from
Singapore to the UK and then re-establish another remote RAID1 store.

delete is an alias of remove, so I added a new disk (devid 3) and
proceeded to run:
btrfs device delete /dev/sdc1 /mnt/raid1 (devid 1)


nuc:~$ uname -a
Linux nuc 4.5.4-1-ARCH #1 SMP PREEMPT Wed May 11 22:21:28 CEST 2016
x86_64 GNU/Linux
nuc:~$   btrfs --version
btrfs-progs v4.5.3
nuc:~$ sudo btrfs fi show /mnt/raid1/
Label: 'extraid1'  uuid: 5cab2a4a-e282-4931-b178-bec4c73cdf77
Total devices 2 FS bytes used 776.56GiB
devid2 size 931.48GiB used 778.03GiB path /dev/sdb1
devid3 size 1.82TiB used 778.03GiB path /dev/sdd1

nuc:~$ sudo btrfs fi df /mnt/raid1/
Data, RAID1: total=775.00GiB, used=774.39GiB
System, RAID1: total=32.00MiB, used=144.00KiB
Metadata, RAID1: total=3.00GiB, used=2.17GiB
GlobalReserve, single: total=512.00MiB, used=0.00B


First off, I was expecting btrfs to release the drive pretty much
immediately. The command took about half a day to complete. I watched
`btrfs fi show` to see size of devid 1 (the one I am trying to remove)
to be zero and to see used space slowly go down whilst used space of
devid 3 (the new disk) slowly go up.
If you're using multiple devices, BTRFS has to move any data off of the 
device being removed onto other devices in the FS before it can remove 
it.  Not doing so runs the risk of another disk failing before the 
balance that would otherwise be needed afterwards to maintain 
replication profiles completes, thus causing a loss of data.


Secondly and most importantly my /dev/sdc1 can't be mounted now anymore.
Why?
Deleting the device from the array removes the data from it (as 
mentioned above), and wipes all BTRFS specific signatures as well.


sudo mount -t btrfs /dev/sdc1 /mnt/test/
mount: wrong fs type, bad option, bad superblock on /dev/sdc1,
   missing codepage or helper program, or other error

   In some cases useful info is found in syslog - try
   dmesg | tail or so.

There is nothing in dmesg nor my journal. I wasn't expecting my drive to
be rendered useless on removing or am I missing something?
If your getting such a message and there's nothing in dmesg, then it's 
because there's no filesystem of the requested type on the device you 
specified.


nuc:~$ sudo fdisk -l /dev/sdc
Disk /dev/sdc: 931.5 GiB, 1000204885504 bytes, 1953525167 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 33553920 bytes
Disklabel type: gpt
Disk identifier: 19938642-3B10-4220-BF99-3E12AF1D1CF6

Device StartEndSectors   Size Type
/dev/sdc1   2048 1953525133 1953523086 931.5G Linux filesystem

On #btrfs IRC channel I'm told:

 hendry: breaking multi-disk filesystems in half is not a
recommended way to do "normal operations" :D
Which is exactly correct.  Trying to split off a single device the way 
you want is extremely dangerous and risks breaking the source filesystem 
as well as not having a complete copy of the data on the receiving end.


I'm still keen to take a TB on a flight with me the day after tomorrow.
What is the recommended course of action? Recreate a mkfs.btrfs on
/dev/sdc1 and send data to it from /mnt/raid1?
Based on the fact that you appear to want to carry a disk to copy data 
more quickly than over then internet, then what you've already done plus 
this is the correct way to do it.


Still I hope the experience could be improved to remove a disk sanely.
It did exactly what you told it to, it removed the device from the 
array.  The misunderstanding here is as to what that means.  Removing a 
disk from a BTRFS filesystem means that you can't mount that disk 
anymore, and for all intents and purposes, it is now blank media (in 
practice, it only wipes the superblocks and some of the metadata, but 
that's irrelevant to what your asking, as you can't recover the 
filesystem from what's left).

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

btrfs device delete /dev/sdc1 /mnt/raid1 user experience

[Kai Hendry]

Hi there,


I planned to remove one of my disks, so that I can take it from
Singapore to the UK and then re-establish another remote RAID1 store.

delete is an alias of remove, so I added a new disk (devid 3) and
proceeded to run:
btrfs device delete /dev/sdc1 /mnt/raid1 (devid 1)


nuc:~$ uname -a
Linux nuc 4.5.4-1-ARCH #1 SMP PREEMPT Wed May 11 22:21:28 CEST 2016
x86_64 GNU/Linux
nuc:~$   btrfs --version
btrfs-progs v4.5.3
nuc:~$ sudo btrfs fi show /mnt/raid1/
Label: 'extraid1'  uuid: 5cab2a4a-e282-4931-b178-bec4c73cdf77
Total devices 2 FS bytes used 776.56GiB
devid2 size 931.48GiB used 778.03GiB path /dev/sdb1
devid3 size 1.82TiB used 778.03GiB path /dev/sdd1

nuc:~$ sudo btrfs fi df /mnt/raid1/
Data, RAID1: total=775.00GiB, used=774.39GiB
System, RAID1: total=32.00MiB, used=144.00KiB
Metadata, RAID1: total=3.00GiB, used=2.17GiB
GlobalReserve, single: total=512.00MiB, used=0.00B


First off, I was expecting btrfs to release the drive pretty much
immediately. The command took about half a day to complete. I watched
`btrfs fi show` to see size of devid 1 (the one I am trying to remove)
to be zero and to see used space slowly go down whilst used space of
devid 3 (the new disk) slowly go up.

Secondly and most importantly my /dev/sdc1 can't be mounted now anymore.
Why?

sudo mount -t btrfs /dev/sdc1 /mnt/test/
mount: wrong fs type, bad option, bad superblock on /dev/sdc1,
   missing codepage or helper program, or other error

   In some cases useful info is found in syslog - try
   dmesg | tail or so.

There is nothing in dmesg nor my journal. I wasn't expecting my drive to
be rendered useless on removing or am I missing something?

nuc:~$ sudo fdisk -l /dev/sdc
Disk /dev/sdc: 931.5 GiB, 1000204885504 bytes, 1953525167 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 33553920 bytes
Disklabel type: gpt
Disk identifier: 19938642-3B10-4220-BF99-3E12AF1D1CF6

Device StartEndSectors   Size Type
/dev/sdc1   2048 1953525133 1953523086 931.5G Linux filesystem

On #btrfs IRC channel I'm told:

 hendry: breaking multi-disk filesystems in half is not a
recommended way to do "normal operations" :D

I'm still keen to take a TB on a flight with me the day after tomorrow.
What is the recommended course of action? Recreate a mkfs.btrfs on
/dev/sdc1 and send data to it from /mnt/raid1?

Still I hope the experience could be improved to remove a disk sanely.

Many thanks,
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html