Hi, some news from the coal mine...
Le 17/03/2017 à 11:03, Lionel Bouton a écrit : > [...] > I'm considering trying to use a 4 week old snapshot of the device to > find out if it was corrupted or not instead. It will still be a pain if > it works but rsync for less than a month of data is at least an order of > magnitude faster than a full restore. btrfs check -p /dev/sdb is running on this 4 week old snapshot. The extents check passed without any error, it is currently checking the free space (and it's just done while I was writing this and is doing fs roots). I'm not sure of the list of checks it performs. I assume the free space^H... fs roots can't be much longer than the rest (on a ~13TB of 20TB used filesystem with ~ 10 million files and half a dozen subvolumes). It took less than an hour to check extents. I'll give it another hour and stop it if its not done : it's already passing stages than the live data couldn't get to. I may be wrong but I suspect Ceph is innocent of any wrong-doing here : I think there's a high probability that if Ceph could corrupt its data in our configuration the snapshot would have been corrupted too (most of its data is shared with the live data). I wonder if QEMU or the VM kernel managed to transform IO timeouts (which clearly happened below Ceph and were passed to the VM in many instances) into garbage reads which ended in garbage writes. If it isn't in QEMU and happened in the kernel this was with 4.1.15 so it might be a corrected kernel bug in either the block or fs layers. I'm not especially ecstatic at the prospect of testing this behavior again but I will automate more Ceph snapshots in the future (and the VM is now on 4.9.6). Best regards, Lionel -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
