[Bug 320638] Re: hot-add/remove in mixed (IDE/SATA/USB/SD-card/...) RAIDs with device mapper on top => data corruption (bio too big device md0 (248 > 240))
This still isn't fixed as best I can follow. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/320638 Title: hot-add/remove in mixed (IDE/SATA/USB/SD-card/...) RAIDs with device mapper on top => data corruption (bio too big device md0 (248 > 240)) To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/320638/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 320638] Re: hot-add/remove in mixed (IDE/SATA/USB/SD-card/...) RAIDs with device mapper on top => data corruption (bio too big device md0 (248 > 240))
** Changed in: linux Status: Confirmed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/320638 Title: hot-add/remove in mixed (IDE/SATA/USB/SD-card/...) RAIDs with device mapper on top => data corruption (bio too big device md0 (248 > 240)) To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/320638/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 320638] Re: hot-add/remove in mixed (IDE/SATA/USB/SD-card/...) RAIDs with device mapper on top = data corruption (bio too big device md0 (248 240))
@Phillip Susi / comment #23: Did you actually read what I wrote? :) I was *NOT* advocating backup by having multiple RAID disks constantly connected to the array and in sync. It is completely obvious to me that a hot running copy of data is NOT a backup. I was advocating the following procedure: 1. Connect disk 2. Wait until it is synced into the array. 3. Shutdown the machine 4. *DISCONNECT* the disc from the machine and consider the completely offline disk as a backup. This is a backup because the disk is physically disconnected from the machine. It is much better than a rsync/cp, because it provides a *coherent* copy since all modifications to the data which happen during the copying process are also applied on the backup. With rsync/cp, files which are modified *after* they have already been copied are not up to date in the backup, and for applications which store data in multiple files (which *many* programs do), their data would be corrupt in such a case. It is relevant to this bugtracker entry because it shows that using multiple kinds of disks in a RAID1, such as SATA+USB, is a common desire and not some exotic border case. Or can you name any other kind of non-exotic, non-beta (such as btrfs) backup mechanism which can copy data while the system is in use without breaking its coherency? :) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/320638 Title: hot-add/remove in mixed (IDE/SATA/USB/SD-card/...) RAIDs with device mapper on top = data corruption (bio too big device md0 (248 240)) To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/320638/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 320638] Re: hot-add/remove in mixed (IDE/SATA/USB/SD-card/...) RAIDs with device mapper on top = data corruption (bio too big device md0 (248 240))
RAID is *not* a backup solution. If you delete or overwrite a file, then it's done on both disks, so you can't recover. If you want a rapid and coherent backup, use LVM and take a snapshot and back that up. Also note that this commentary really isn't helping to fix the bug. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/320638 Title: hot-add/remove in mixed (IDE/SATA/USB/SD-card/...) RAIDs with device mapper on top = data corruption (bio too big device md0 (248 240)) To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/320638/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 320638] Re: hot-add/remove in mixed (IDE/SATA/USB/SD-card/...) RAIDs with device mapper on top = data corruption (bio too big device md0 (248 240))
Also, keep on mind that the most commonly used personal computers nowadays don't even *support* adding multiple disks of the same type: Laptops. They only have one HD slot, so I *must* use USB to attach the second. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/320638 Title: hot-add/remove in mixed (IDE/SATA/USB/SD-card/...) RAIDs with device mapper on top = data corruption (bio too big device md0 (248 240)) To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/320638/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 320638] Re: hot-add/remove in mixed (IDE/SATA/USB/SD-card/...) RAIDs with device mapper on top = data corruption (bio too big device md0 (248 240))
What you people are forgetting is that RAID1 is in fact the PERFECT backup solution: It takes a low level copy of the whole system while the system is *running*, and as opposed to cp/rsync, the copy is *coherent*: Programmers do NOT design software to be robust against their files being randomly copied one after another by an external cp/rsync. How is this even supposed to work? If you copy file A, then file B, nothing guarantees that the software does not modify one of them in between your copying in a way which makes the files incoherent / breaks their compatibility. And with regards to backup, we're talking about copying whole operating systems, consisting of *thousands* of programs. I would say the probability among thousands of programs is 100% that at least one of them will have its data corrupted if you copy its files using cp/rsync while it is running. As opposed to that, a RAID1 is always in complete coherence if you shut down the system before you remove the disk. There is no mismatching data in multiple files. You can immediately boot up again, so backup takes 1 minute. You could of course take offline the system for cp/rsync as well to get coherent data, but that will give you *hours* of downtime because the copying needs to happen while the system is offline. RAID1 can copy while it is running! Overall, this bug is the worst issue in software design I've encountered in years, and I'm almost screaming :( I have spent over 20 hours migrating my machines to RAID1 so I can reduce the backup procedure from 5 hours to 5 minutes, and now it just doesn't WORK. This is infuriating :( Can someone *please* come up with a fix / workaround? I would be willing to pay a bounty of 50 EUR in Bitcoin for this to be fixed. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/320638 Title: hot-add/remove in mixed (IDE/SATA/USB/SD-card/...) RAIDs with device mapper on top = data corruption (bio too big device md0 (248 240)) To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/320638/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 320638] Re: hot-add/remove in mixed (IDE/SATA/USB/SD-card/...) RAIDs with device mapper on top = data corruption (bio too big device md0 (248 240))
** Changed in: linux Status: Fix Released = Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/320638 Title: hot-add/remove in mixed (IDE/SATA/USB/SD-card/...) RAIDs with device mapper on top = data corruption (bio too big device md0 (248 240)) To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/320638/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 320638] Re: hot-add/remove in mixed (IDE/SATA/USB/SD-card/...) RAIDs with device mapper on top = data corruption (bio too big device md0 (248 240))
The resolution of fix released is incorrect: the kernel bug is still present. The debian bug was closed due to age rather due to being fixed. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/320638 Title: hot-add/remove in mixed (IDE/SATA/USB/SD-card/...) RAIDs with device mapper on top = data corruption (bio too big device md0 (248 240)) To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/320638/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 320638] Re: hot-add/remove in mixed (IDE/SATA/USB/SD-card/...) RAIDs with device mapper on top = data corruption (bio too big device md0 (248 240))
** Changed in: linux Status: Confirmed = Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/320638 Title: hot-add/remove in mixed (IDE/SATA/USB/SD-card/...) RAIDs with device mapper on top = data corruption (bio too big device md0 (248 240)) To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/320638/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 320638] Re: hot-add/remove in mixed (IDE/SATA/USB/SD-card/...) RAIDs with device mapper on top = data corruption (bio too big device md0 (248 240))
I don't know, how it behaves with USB3. Same data corruption as well, unfortunately. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/320638 Title: hot-add/remove in mixed (IDE/SATA/USB/SD-card/...) RAIDs with device mapper on top = data corruption (bio too big device md0 (248 240)) To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/320638/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 320638] Re: hot-add/remove in mixed (IDE/SATA/USB/SD-card/...) RAIDs with device mapper on top = data corruption (bio too big device md0 (248 240))
** Description changed: Problem: md changes max_sector setting of an already running and busy md device, when a (hotplugable) device is added or removed. However, the device mapper and filesystem layer on top of the raid can not (always?) cope with that. Observations: * bio too big device mdX (248 240) messages in the syslog * read/write errors (some dropped silently, no noticable errors reported during operation, until things like dhcpclient looses its IP etc.) Expected: Adding and removing members to running raids (hotplugging) should not change the raid device characteristics. If the new member supports only smaller max_sector values, buffer and split the data steam, until the raid device can be set up from a clean state with a more appropriate max_sector value. To avoid buffering and splitting in the future, md could save the smallest max_sector value of the known members in the superblock, and use that when setting up the raid even if that member is not present. - Note: This is reproducible in much more common scenarios as the original reporter had (e.g. --add a USB (3.0 these days) drive to an already running SATA raid1 and grow the number of devices). + Note: This is reproducible in much more common scenarios as the original + reporter had (e.g. --add a USB (3.0 these days) drive to an already + running SATA raid1 and grow the number of devices). + + Fix: + Upsteam has no formal bug tracking, but a mailing list. The response was that finally this needs to be fixed [outside of mdadm] by cleaning up the bio path so that big bios are split by the device that needs the split, not be the fs sending the bio. + + However, in the meantime mdadm needs to saveguard against the date + corruption: + + [The mdadm] fix is to reject the added device [if] its limits are + too low. + + Good Idea to avoid the data corruption. MD could save the + max_sectors default limit for arrays. If the array is modified and the new + limit gets smaller, postpone the sync until the next assembe/restart. + + And of course print a message if postponing, that explains when --force would be save. + What ever that would be: no block device abstraction layer (device mapper, lvm, luks,...) + between an unmounted? ext, fat?, ...? filesystem and md? + + As upsteam does not do public bug tracking, the status and rememberence + of this need remains unsure though. + + --- This is on a MSI Wind U100 and I've got the following stack running: HDD SD card (USB card reader) - RAID1 - LUKS - LVM - Reiser Whenever I remove the HDD from the Raid1 mdadm /dev/md0 --fail /dev/sda2 mdadm /dev/md0 --remove /dev/sda2) for powersaving reasons, I cannot run any apt related tools. sudo apt-get update [...] Hit http://de.archive.ubuntu.com intrepid-updates/multiverse Sources Reading package lists... Error! E: Read error - read (5 Input/output error) E: The package lists or status file could not be parsed or opened. Taking a look at the kernel log shows (and many more above): dmesg|tail [ 9479.330550] bio too big device md0 (248 240) [ 9479.331375] bio too big device md0 (248 240) [ 9479.332182] bio too big device md0 (248 240) [ 9611.980294] bio too big device md0 (248 240) [ 9742.929761] bio too big device md0 (248 240) [ 9852.932001] bio too big device md0 (248 240) [ 9852.935395] bio too big device md0 (248 240) [ 9852.938064] bio too big device md0 (248 240) [ 9853.081046] bio too big device md0 (248 240) [ 9853.081688] bio too big device md0 (248 240) $ sudo mdadm --detail /dev/md0 /dev/md0: Version : 00.90 Creation Time : Tue Jan 13 11:25:57 2009 Raid Level : raid1 Array Size : 3871552 (3.69 GiB 3.96 GB) Used Dev Size : 3871552 (3.69 GiB 3.96 GB) Raid Devices : 2 Total Devices : 1 Preferred Minor : 0 Persistence : Superblock is persistent Intent Bitmap : Internal Update Time : Fri Jan 23 21:47:35 2009 State : active, degraded Active Devices : 1 Working Devices : 1 Failed Devices : 0 Spare Devices : 0 UUID : 89863068:bc52a0c0:44a5346e:9d69deca (local to host m-twain) Events : 0.8767 Number Major Minor RaidDevice State 0 000 removed 1 8 171 active sync writemostly /dev/sdb1 $ sudo ubuntu-bug -p linux-meta dpkg-query: failed in buffer_read(fd): copy info file `/var/lib/dpkg/status': Input/output error dpkg-query: failed in buffer_read(fd): copy info file `/var/lib/dpkg/status': Input/output error [...] Will provide separate attachements. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/320638 Title: hot-add/remove in mixed (IDE/SATA/USB/SD-card/...) RAIDs with device mapper on top = data
[Bug 320638] Re: hot-add/remove in mixed (IDE/SATA/USB/SD-card/...) RAIDs with device mapper on top = data corruption (bio too big device md0 (248 240))
I'm using SATA drives for LVM on RAID 1. RAID is configured with internal bitmap, so resyncing is quite fast. In order to do the best kind of backup, I replace my drives to keep backups of complete systems. There is a very good reason for this, since in the case of a complete desaster I simply take the last backup disk and use it in a working hardware. So, sometimes it would be quite useful, to also use some USB external storage to connect a hard drive. But, because of this error, the whole procedure is not possible. I know it works with eSATA, but this error claims always with USB connected drives. Using rsync is not as simple as using a hotplug script that initiates raid resync. I don't know, how it behaves with USB3. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/320638 Title: hot-add/remove in mixed (IDE/SATA/USB/SD-card/...) RAIDs with device mapper on top = data corruption (bio too big device md0 (248 240)) To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/320638/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 320638] Re: hot-add/remove in mixed (IDE/SATA/USB/SD-card/...) RAIDs with device mapper on top = data corruption (bio too big device md0 (248 240))
This is an upstream issue in kernel / handling nested block device (lvm on top of mdadm). It would be interesting to know if the patches to recursively call merge_bvec_fn became a reality nowadays. [1] It's best to ping linux-raid mailing list to query if anything changed with respect to this bug. [1] http://lkml.indiana.edu/hypermail/linux/kernel/0704.1/1008.html ** Changed in: debian-installer (Ubuntu) Status: Confirmed = Invalid ** Changed in: ubiquity (Ubuntu) Status: New = Invalid -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/320638 Title: hot-add/remove in mixed (IDE/SATA/USB/SD-card/...) RAIDs with device mapper on top = data corruption (bio too big device md0 (248 240)) To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/320638/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 320638] Re: hot-add/remove in mixed (IDE/SATA/USB/SD-card/...) RAIDs with device mapper on top = data corruption (bio too big device md0 (248 240))
Never mind, Neil recently replied about the state of the art w.r.t. to this bug: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=624343#106 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/320638 Title: hot-add/remove in mixed (IDE/SATA/USB/SD-card/...) RAIDs with device mapper on top = data corruption (bio too big device md0 (248 240)) To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/320638/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 320638] Re: hot-add/remove in mixed (IDE/SATA/USB/SD-card/...) RAIDs with device mapper on top = data corruption (bio too big device md0 (248 240))
** Description changed: - Note: Bug is also present when hot-plugging USB, Firewire etc. devices. + Problem: md changes max_sector setting of an already running and busy md + device, when a (hotplugable) device is added or removed. However, the a + device mapper and filesystem layer on top of the raid can not (always?) + cope with that. - Also reproducable in much more common usage as originally reported (e.g. --add a USB (3.0 these days) drive to an already running SATA raid1 and grow the number of devices). + Observations: + * bio too big device mdX (248 240) messages in the syslog + * read/write errors (some dropped silently, no noticable errors reported during operation, until things like dhcpclient looses its IP etc.) + + Expected: + Adding and removing members to running raids (hotplugging) does not change the raid device characteristics. If the new member supports only smaller max_sector values, buffer and split the data steam, until the raid device can be set up from a clean state with a more appropriate max_sector value. To avoid buffering and splitting in the future, md could save the smallest max_sector value of the known members in the superblock, and use that when setting up the raid even if that member is not present. + + + Note: This is reproducible in much more common scenarios as the original reporter had (e.g. --add a USB (3.0 these days) drive to an already running SATA raid1 and grow the number of devices). --- This is on a MSI Wind U100 and I've got the following stack running: HDD SD card (USB card reader) - RAID1 - LUKS - LVM - Reiser Whenever I remove the HDD from the Raid1 mdadm /dev/md0 --fail /dev/sda2 mdadm /dev/md0 --remove /dev/sda2) for powersaving reasons, I cannot run any apt related tools. sudo apt-get update [...] Hit http://de.archive.ubuntu.com intrepid-updates/multiverse Sources Reading package lists... Error! E: Read error - read (5 Input/output error) E: The package lists or status file could not be parsed or opened. Taking a look at the kernel log shows (and many more above): dmesg|tail [ 9479.330550] bio too big device md0 (248 240) [ 9479.331375] bio too big device md0 (248 240) [ 9479.332182] bio too big device md0 (248 240) [ 9611.980294] bio too big device md0 (248 240) [ 9742.929761] bio too big device md0 (248 240) [ 9852.932001] bio too big device md0 (248 240) [ 9852.935395] bio too big device md0 (248 240) [ 9852.938064] bio too big device md0 (248 240) [ 9853.081046] bio too big device md0 (248 240) [ 9853.081688] bio too big device md0 (248 240) $ sudo mdadm --detail /dev/md0 /dev/md0: Version : 00.90 Creation Time : Tue Jan 13 11:25:57 2009 Raid Level : raid1 Array Size : 3871552 (3.69 GiB 3.96 GB) Used Dev Size : 3871552 (3.69 GiB 3.96 GB) Raid Devices : 2 Total Devices : 1 Preferred Minor : 0 Persistence : Superblock is persistent Intent Bitmap : Internal Update Time : Fri Jan 23 21:47:35 2009 State : active, degraded Active Devices : 1 Working Devices : 1 Failed Devices : 0 Spare Devices : 0 UUID : 89863068:bc52a0c0:44a5346e:9d69deca (local to host m-twain) Events : 0.8767 Number Major Minor RaidDevice State 0 000 removed 1 8 171 active sync writemostly /dev/sdb1 $ sudo ubuntu-bug -p linux-meta dpkg-query: failed in buffer_read(fd): copy info file `/var/lib/dpkg/status': Input/output error dpkg-query: failed in buffer_read(fd): copy info file `/var/lib/dpkg/status': Input/output error [...] Will provide separate attachements. ** Description changed: Problem: md changes max_sector setting of an already running and busy md - device, when a (hotplugable) device is added or removed. However, the a + device, when a (hotplugable) device is added or removed. However, the device mapper and filesystem layer on top of the raid can not (always?) cope with that. Observations: * bio too big device mdX (248 240) messages in the syslog * read/write errors (some dropped silently, no noticable errors reported during operation, until things like dhcpclient looses its IP etc.) Expected: Adding and removing members to running raids (hotplugging) does not change the raid device characteristics. If the new member supports only smaller max_sector values, buffer and split the data steam, until the raid device can be set up from a clean state with a more appropriate max_sector value. To avoid buffering and splitting in the future, md could save the smallest max_sector value of the known members in the superblock, and use that when setting up the raid even if that member is not present. - Note: This is reproducible in much more common scenarios as the original reporter had (e.g. --add a USB (3.0 these days) drive to an
[Bug 320638] Re: hot-add/remove in mixed (IDE/SATA/USB/SD-card/...) RAIDs with device mapper on top = data corruption (bio too big device md0 (248 240))
** Also affects: mdadm Importance: Undecided Status: New ** Changed in: mdadm Status: New = Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/320638 Title: hot-add/remove in mixed (IDE/SATA/USB/SD-card/...) RAIDs with device mapper on top = data corruption (bio too big device md0 (248 240)) To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/320638/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
