Re: Two partitionless BTRFS drives no longer seen as containing BTRFS filesystem
> None of your super blocks has correct magic. I take it this applies to both drives? > This means either your whole disk get corrupted, or something introduced > some offset. > > Please try the following commands to dump more data around super blocks, > so we could be able to find the possible offset: > > # dd if=/dev/sdb of=possible_sb_range.raw bs=1M count=128 > # grep -obUaP "\x5F\x42\x48\x52\x66\x53\x5F\x4D" possible_sb_range. > > For a valid btrfs without any offset, the result should look like: > 65600:_BHRfS_M > 67108928:_BHRfS_M # dd if=/dev/sdc of=possible_sb_range.sdc.raw bs=1M count=128 128+0 records in 128+0 records out 134217728 bytes (134 MB, 128 MiB) copied, 0.737479 s, 182 MB/s # dd if=/dev/sdb of=possible_sb_range.sdb.raw bs=1M count=128 128+0 records in 128+0 records out 134217728 bytes (134 MB, 128 MiB) copied, 0.726327 s, 185 MB/s # grep -obUaP "\x5F\x42\x48\x52\x66\x53\x5F\x4D" possible_sb_range.sdb.raw # grep -obUaP "\x5F\x42\x48\x52\x66\x53\x5F\x4D" possible_sb_range.sdc.raw Both return nothing. Both drives pass S.M.A.R.T. testing so I'd have to think the corruption stems from some kind of offset rather than random corruption. They may have accidentally been inserted into a Windows machine (but not partitioned or formatted). Could this be a likely cause?
Re: Two partitionless BTRFS drives no longer seen as containing BTRFS filesystem
On 2018/10/7 下午2:10, evan d wrote: >> Please try "btrfs ins dump-super -fFa" on these two disks. >> >> If it's only the primary superblock corrupted, the backup should be good. >> >> If backup is also corrupted, either it has some offset or the whole data >> is corrupted. > > # btrfs ins dump-super -fFa /dev/sdb > superblock: bytenr=65536, device=/dev/sdb > - > csum_type 0 (crc32c) > csum_size 4 > csum 0x [DON'T MATCH] > bytenr 0 > flags 0x0 > magic [DON'T MATCH] [snip] > superblock: bytenr=274877906944, device=/dev/sdb > - > csum_type 26294 (INVALID) > csum_size 32 > csum 0x05401fa3a3e8cd64075ce9fdbb9e60a01d58061a3cff3bc0235d18912ab755a2 > [UNKNOWN CSUM TYPE OR SIZE] > bytenr 7401042280310172376 > flags 0x5a8a759265673a05 > ( WRITTEN | > METADUMP | > unknown flag: 0x5a8a759065673a04 ) > magic .~...6.. [DON'T MATCH] [snip] None of your super blocks has correct magic. This means either your whole disk get corrupted, or something introduced some offset. Please try the following commands to dump more data around super blocks, so we could be able to find the possible offset: # dd if=/dev/sdb of=possible_sb_range.raw bs=1M count=128 # grep -obUaP "\x5F\x42\x48\x52\x66\x53\x5F\x4D" possible_sb_range. For a valid btrfs without any offset, the result should look like: 65600:_BHRfS_M 67108928:_BHRfS_M If your result doesn't look like this but still has two similar hits, then you could calculate the offset, and use dm-linear to remap the disk and try to recover the fs. Thanks, Qu signature.asc Description: OpenPGP digital signature
Re: Two partitionless BTRFS drives no longer seen as containing BTRFS filesystem
> Please try "btrfs ins dump-super -fFa" on these two disks. > > If it's only the primary superblock corrupted, the backup should be good. > > If backup is also corrupted, either it has some offset or the whole data > is corrupted. # btrfs ins dump-super -fFa /dev/sdb superblock: bytenr=65536, device=/dev/sdb - csum_type 0 (crc32c) csum_size 4 csum 0x [DON'T MATCH] bytenr 0 flags 0x0 magic [DON'T MATCH] fsid ---- label generation 0 root 0 sys_array_size 0 chunk_root_generation 0 root_level 0 chunk_root 0 chunk_root_level 0 log_root 0 log_root_transid 0 log_root_level 0 total_bytes 0 bytes_used 0 sectorsize 0 nodesize 0 leafsize (deprecated) 0 stripesize 0 root_dir 0 num_devices 0 compat_flags 0x0 compat_ro_flags 0x0 incompat_flags 0x0 cache_generation 0 uuid_tree_generation 0 dev_item.uuid ---- dev_item.fsid ---- [match] dev_item.type 0 dev_item.total_bytes 0 dev_item.bytes_used 0 dev_item.io_align 0 dev_item.io_width 0 dev_item.sector_size 0 dev_item.devid 0 dev_item.dev_group 0 dev_item.seek_speed 0 dev_item.bandwidth 0 dev_item.generation 0 sys_chunk_array[2048]: backup_roots[4]: superblock: bytenr=67108864, device=/dev/sdb - csum_type 29777 (INVALID) csum_size 32 csum 0xdfa25558dc616d7f36b287aa0081da91b78ec4910ba082b8d807708dcf608c91 [UNKNOWN CSUM TYPE OR SIZE] bytenr 18068171547813619898 flags 0xbaca4264c914fdc ( METADUMP | METADUMP_V2 | unknown flag: 0xbaca4204c914fdc ) magic Y...FE./ [DON'T MATCH] fsid 870d0bb4-65de-453b-2592-88b2cda7c38b label >.<...@...B%...<.N..S.*.>^.p.x.v..o..<).u..B}./...B..3:8..&}-k...o6..Y..R..'o..G.lG.-...$'S!.h.-...".*.S..xf..@d...W..*..w..V.)..q..|.pgt.L...z.i...(..3Ar...;B.j.]:I.X...=x.(u.<..2U..0 .n..^}...^..Y.a#.X...G..d.. generation 2272432616352171010 root 6354703493070726827 sys_array_size 2955003080 chunk_root_generation 6888771666932949452 root_level 123 chunk_root 1494381076279922822 chunk_root_level 127 log_root 2134514170013133221 log_root_transid 9424889336279151653 log_root_level 153 total_bytes 2681859058010843869 bytes_used 16135710043447639801 sectorsize 1588161000 nodesize 3303803296 leafsize (deprecated) 1829755703 stripesize 606246139 root_dir 1342036912498050847 num_devices 17153780124435501376 compat_flags 0xf49b2c3a96602d1e compat_ro_flags 0x8bec2f0d81d2ef13 ( FREE_SPACE_TREE | FREE_SPACE_TREE_VALID | unknown flag: 0x8bec2f0d81d2ef10 ) incompat_flags 0x1af3f95d789a706e ( DEFAULT_SUBVOL | MIXED_GROUPS | COMPRESS_LZO | BIG_METADATA | EXTENDED_IREF | unknown flag: 0x1af3f95d789a7000 ) cache_generation 733155394646473 uuid_tree_generation 6538806993670512709 dev_item.uuid c1494bde-9546-5ef1-5d08-e259707d06d3 dev_item.fsid 584a42f5-4f79-58cc-3f5e-bdd9733b94d3 [DON'T MATCH] dev_item.type 15668929679594826040 dev_item.total_bytes 1644623768791993611 dev_item.bytes_used 13437937382560806996 dev_item.io_align 449600477 dev_item.io_width 3390996026 dev_item.sector_size 2580293710 dev_item.devid 8727228223548828735 dev_item.dev_group 3143108434 dev_item.seek_speed 99 dev_item.bandwidth 34 dev_item.generation 15553054142077970558 sys_chunk_array[2048]: ERROR: sys_array_size 2955003080 shouldn't exceed 2048 bytes backup_roots[4]: backup 0: backup_tree_root: 15696464816120474628 gen: 16022257131800929882 level: 86 backup_chunk_root: 301766055038840723 gen: 16810756911197712753 level: 114 backup_extent_root: 6651488794176833875 gen: 10776594467847637718 level: 60 backup_fs_root: 1826104903792017114 gen: 3329223824114931446 level: 159 backup_dev_root: 11721506207622158585 gen: 10455859429120851009 level: 198 backup_csum_root: 5686172936498246011 gen: 5319088827707453088 level: 168 backup_total_bytes: 7660601670332883006 backup_bytes_used: 3723313713264611767 backup_num_devices: 1913069816786984281 backup 1: backup_tree_root: 9129110729577497674 gen: 17870813394716935947 level: 45 backup_chunk_root: 3169491491968745044 gen: 17348548561480407615 level: 93 backup_extent_root: 5781159137873655776 gen: 3348348558872496210 level: 123 backup_fs_root: 66326293521237128 gen: 16098559310782853786 level: 157 backup_dev_root: 10010186234695580749 gen: 5427645709246451749 level: 214 backup_csum_root: 3481616510852897026 gen: 3557794445033232028 level: 233 backup_total_bytes: 14437518517144737363 backup_bytes_used: 17463569409584801738 backup_num_devices: 3997309709667846939 backup 2: backup_tree_root: 10726132636215357139 gen: 12411364641008183307 level: 88 backup_chunk_root: 15701704192942804973 gen: 12075216484835399161 level: 115 backup_extent_root: 2480766121519302854 gen: 14058965640461957484 level: 149 backup_fs_root: 9763730962528489871 gen: 7584542795525942005 level: 178 backup_dev_root: 11165845436133270817 gen: 13967062440412348994 level: 236 backup_csum_root: 617189674839628565
Re: Two partitionless BTRFS drives no longer seen as containing BTRFS filesystem
> Did you try a btrfs device scan ? Tried it, it returns nothing.
btrfs-progs 32/64 bits mode unconsistency
Hi, If the 'btrfs' executable is 32 bits and the kernel is 64 bits, then 'btrfs replace status' and 'btrfs replace start' commands will fail with error message: ERROR: ioctl(DEV_REPLACE_STATUS) failed on "/mnt/data1":Inappropriate ioctl for device Other commands seem to work properly. Regards, Hubert Tonneau
Re: Two partitionless BTRFS drives no longer seen as containing BTRFS filesystem
On 2018/10/7 上午7:23, evan d wrote: > I have two hard drives that were never partitioned, but set up as two > independent BRTFS filesystems. Both drives were used in the same > machine running Arch Linux and the drives contain(ed) largely static > data. > > I decommissioned the machine they were originally used in and on > installing in a newer Arch build found that BRTFS reported no > filesystem on either of the drives. Please try "btrfs ins dump-super -fFa" on these two disks. If it's only the primary superblock corrupted, the backup should be good. If backup is also corrupted, either it has some offset or the whole data is corrupted. Thanks, Qu > > uname -a: > Linux z87i-pro 4.18.9-arch1-1-ARCH #1 SMP PREEMPT Wed Sep 19 21:19:17 > UTC 2018 x86_64 GNU/Linux > > btrfs --version: btrfs-progs v4.17.1 > btrfs fi show: returns no data > > parted -l: > Error: /dev/sdb: unrecognised disk label > Model: ATA WDC WD60EFRX-68M (scsi) > Disk /dev/sdb: 6001GB > Sector size (logical/physical): 512B/4096B > Partition Table: unknown > Disk Flags: > > gdisk /dev/sdb - l: > Error: /dev/sdc: unrecognised disk label > Model: ATA WDC WD60EFRX-68M (scsi) > Disk /dev/sdc: 6001GB > Sector size (logical/physical): 512B/4096B > Partition Table: unknown > Disk Flags: > > gdisk /dev/sdb -l > GPT fdisk (gdisk) version 1.0.4 > > Caution: invalid main GPT header, but valid backup; regenerating main header > from backup! > > Caution! After loading partitions, the CRC doesn't check out! > Warning: Invalid CRC on main header data; loaded backup partition table. > Warning! Main and backup partition tables differ! Use the 'c' and 'e' options > on the recovery & transformation menu to examine the two tables. > > Warning! One or more CRCs don't match. You should repair the disk! > Main header: ERROR > Backup header: OK > Main partition table: ERROR > Backup partition table: ERROR > > Partition table scan: > MBR: protective > BSD: not present > APM: not present > GPT: damaged > > > Caution: Found protective or hybrid MBR and corrupt GPT. Using GPT, but disk > verification and recovery are STRONGLY recommended. > > Warning! Main partition table overlaps the first partition by 33 blocks! > You will need to delete this partition or resize it in another utility. > Disk /dev/sdb: 11721045168 sectors, 5.5 TiB > Model: WDC WD60EFRX-68M > Sector size (logical/physical): 512/4096 bytes > Disk identifier (GUID): FB26D91F-0709-11E7-B298-3464A99AF244 > Partition table holds up to 128 entries > Main partition table begins at sector 2 and ends at sector 33 > First usable sector is 34, last usable sector is 11721045134 > Partitions will be aligned on 8-sector boundaries > Total free space is 11721045101 sectors (5.5 TiB) > > Number Start (sector)End (sector) Size Code Name > 69 1 6 3.0 KiB 犀읭퇸䵧羒⌦盤䧔Ẑ坔 > > gdisk /dev/sdc -l: > GPT fdisk (gdisk) version 1.0.4 > > Caution: invalid main GPT header, but valid backup; regenerating main header > from backup! > > Caution! After loading partitions, the CRC doesn't check out! > Warning: Invalid CRC on main header data; loaded backup partition table. > Warning! Main and backup partition tables differ! Use the 'c' and 'e' options > on the recovery & transformation menu to examine the two tables. > > Warning! One or more CRCs don't match. You should repair the disk! > Main header: ERROR > Backup header: OK > Main partition table: ERROR > Backup partition table: ERROR > > Partition table scan: > MBR: protective > BSD: not present > APM: not present > GPT: damaged > > > Caution: Found protective or hybrid MBR and corrupt GPT. Using GPT, but disk > verification and recovery are STRONGLY recommended. > > Warning! Main partition table overlaps the first partition by 33 blocks! > You will need to delete this partition or resize it in another utility. > Disk /dev/sdc: 11721045168 sectors, 5.5 TiB > Model: WDC WD60EFRX-68M > Sector size (logical/physical): 512/4096 bytes > Disk identifier (GUID): FC86A481-0709-11E7-B298-3464A99AF244 > Partition table holds up to 128 entries > Main partition table begins at sector 2 and ends at sector 33 > First usable sector is 34, last usable sector is 11721045134 > Partitions will be aligned on 8-sector boundaries > Total free space is 11721045101 sectors (5.5 TiB) > > Number Start (sector)End (sector) Size Code Name > 69 1 6 3.0 KiB 윜㺗ᣲ䌟犔熵䕭㶖Ẑ坔 > > > Any ideas what's happened here seeing as both drives appear to suffer > the same symptoms and what I can do to attempt a recovery? > signature.asc Description: OpenPGP digital signature
Re: Two partitionless BTRFS drives no longer seen as containing BTRFS filesystem
On 2018-10-06 07:23 PM, evan d wrote: > I have two hard drives that were never partitioned, but set up as two > independent BRTFS filesystems. Both drives were used in the same > machine running Arch Linux and the drives contain(ed) largely static > data. > > I decommissioned the machine they were originally used in and on > installing in a newer Arch build found that BRTFS reported no > filesystem on either of the drives. > > uname -a: > Linux z87i-pro 4.18.9-arch1-1-ARCH #1 SMP PREEMPT Wed Sep 19 21:19:17 > UTC 2018 x86_64 GNU/Linux > > btrfs --version: btrfs-progs v4.17.1 > btrfs fi show: returns no data Did you try a btrfs device scan ? (Normally, that would be done on boot, but depending on how your arch was configured, or if the devices are available early enough in the boot process) <>
Two partitionless BTRFS drives no longer seen as containing BTRFS filesystem
I have two hard drives that were never partitioned, but set up as two independent BRTFS filesystems. Both drives were used in the same machine running Arch Linux and the drives contain(ed) largely static data. I decommissioned the machine they were originally used in and on installing in a newer Arch build found that BRTFS reported no filesystem on either of the drives. uname -a: Linux z87i-pro 4.18.9-arch1-1-ARCH #1 SMP PREEMPT Wed Sep 19 21:19:17 UTC 2018 x86_64 GNU/Linux btrfs --version: btrfs-progs v4.17.1 btrfs fi show: returns no data parted -l: Error: /dev/sdb: unrecognised disk label Model: ATA WDC WD60EFRX-68M (scsi) Disk /dev/sdb: 6001GB Sector size (logical/physical): 512B/4096B Partition Table: unknown Disk Flags: gdisk /dev/sdb - l: Error: /dev/sdc: unrecognised disk label Model: ATA WDC WD60EFRX-68M (scsi) Disk /dev/sdc: 6001GB Sector size (logical/physical): 512B/4096B Partition Table: unknown Disk Flags: gdisk /dev/sdb -l GPT fdisk (gdisk) version 1.0.4 Caution: invalid main GPT header, but valid backup; regenerating main header from backup! Caution! After loading partitions, the CRC doesn't check out! Warning: Invalid CRC on main header data; loaded backup partition table. Warning! Main and backup partition tables differ! Use the 'c' and 'e' options on the recovery & transformation menu to examine the two tables. Warning! One or more CRCs don't match. You should repair the disk! Main header: ERROR Backup header: OK Main partition table: ERROR Backup partition table: ERROR Partition table scan: MBR: protective BSD: not present APM: not present GPT: damaged Caution: Found protective or hybrid MBR and corrupt GPT. Using GPT, but disk verification and recovery are STRONGLY recommended. Warning! Main partition table overlaps the first partition by 33 blocks! You will need to delete this partition or resize it in another utility. Disk /dev/sdb: 11721045168 sectors, 5.5 TiB Model: WDC WD60EFRX-68M Sector size (logical/physical): 512/4096 bytes Disk identifier (GUID): FB26D91F-0709-11E7-B298-3464A99AF244 Partition table holds up to 128 entries Main partition table begins at sector 2 and ends at sector 33 First usable sector is 34, last usable sector is 11721045134 Partitions will be aligned on 8-sector boundaries Total free space is 11721045101 sectors (5.5 TiB) Number Start (sector)End (sector) Size Code Name 69 1 6 3.0 KiB 犀읭퇸䵧羒⌦盤䧔Ẑ坔 gdisk /dev/sdc -l: GPT fdisk (gdisk) version 1.0.4 Caution: invalid main GPT header, but valid backup; regenerating main header from backup! Caution! After loading partitions, the CRC doesn't check out! Warning: Invalid CRC on main header data; loaded backup partition table. Warning! Main and backup partition tables differ! Use the 'c' and 'e' options on the recovery & transformation menu to examine the two tables. Warning! One or more CRCs don't match. You should repair the disk! Main header: ERROR Backup header: OK Main partition table: ERROR Backup partition table: ERROR Partition table scan: MBR: protective BSD: not present APM: not present GPT: damaged Caution: Found protective or hybrid MBR and corrupt GPT. Using GPT, but disk verification and recovery are STRONGLY recommended. Warning! Main partition table overlaps the first partition by 33 blocks! You will need to delete this partition or resize it in another utility. Disk /dev/sdc: 11721045168 sectors, 5.5 TiB Model: WDC WD60EFRX-68M Sector size (logical/physical): 512/4096 bytes Disk identifier (GUID): FC86A481-0709-11E7-B298-3464A99AF244 Partition table holds up to 128 entries Main partition table begins at sector 2 and ends at sector 33 First usable sector is 34, last usable sector is 11721045134 Partitions will be aligned on 8-sector boundaries Total free space is 11721045101 sectors (5.5 TiB) Number Start (sector)End (sector) Size Code Name 69 1 6 3.0 KiB 윜㺗ᣲ䌟犔熵䕭㶖Ẑ坔 Any ideas what's happened here seeing as both drives appear to suffer the same symptoms and what I can do to attempt a recovery? [0.00] microcode: microcode updated early to revision 0x25, date = 2018-04-02 [0.00] Linux version 4.18.9-arch1-1-ARCH (builduser@heftig-19946) (gcc version 8.2.1 20180831 (GCC)) #1 SMP PREEMPT Wed Sep 19 21:19:17 UTC 2018 [0.00] Command line: initrd=\intel-ucode.img initrd=\initramfs-linux.img root=PARTUUID=7f3a7108-c5bc-40dd-856c-dea1ae6ef2a3 rw [0.00] KERNEL supported cpus: [0.00] Intel GenuineIntel [0.00] AMD AuthenticAMD [0.00] Centaur CentaurHauls [0.00] x86/fpu: Supporting XSAVE feature 0x00
Re: [PATCH RESEND] Btrfs: make should_defrag_range() understood compressed extents
вт, 18 сент. 2018 г. в 13:09, Timofey Titovets :
>
> From: Timofey Titovets
>
> Both, defrag ioctl and autodefrag - call btrfs_defrag_file()
> for file defragmentation.
>
> Kernel default target extent size - 256KiB.
> Btrfs progs default - 32MiB.
>
> Both bigger then maximum size of compressed extent - 128KiB.
> That lead to rewrite all compressed data on disk.
>
> Fix that by check compression extents with different logic.
>
> As addition, make should_defrag_range() understood compressed extent type,
> if requested target compression are same as current extent compression type.
> Just don't recompress/rewrite extents.
> To avoid useless recompression of compressed extents.
>
> Signed-off-by: Timofey Titovets
> ---
> fs/btrfs/ioctl.c | 28 +---
> 1 file changed, 25 insertions(+), 3 deletions(-)
>
> diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c
> index a990a9045139..0a5ea1ccc89d 100644
> --- a/fs/btrfs/ioctl.c
> +++ b/fs/btrfs/ioctl.c
> @@ -1142,7 +1142,7 @@ static bool defrag_check_next_extent(struct inode
> *inode, struct extent_map *em)
>
> static int should_defrag_range(struct inode *inode, u64 start, u32 thresh,
>u64 *last_len, u64 *skip, u64 *defrag_end,
> - int compress)
> + int compress, int compress_type)
> {
> struct extent_map *em;
> int ret = 1;
> @@ -1177,8 +1177,29 @@ static int should_defrag_range(struct inode *inode,
> u64 start, u32 thresh,
> * real extent, don't bother defragging it
> */
> if (!compress && (*last_len == 0 || *last_len >= thresh) &&
> - (em->len >= thresh || (!next_mergeable && !prev_mergeable)))
> + (em->len >= thresh || (!next_mergeable && !prev_mergeable))) {
> ret = 0;
> + goto out;
> + }
> +
> +
> + /*
> +* Try not recompress compressed extents
> +* thresh >= BTRFS_MAX_UNCOMPRESSED will lead to
> +* recompress all compressed extents
> +*/
> + if (em->compress_type != 0 && thresh >= BTRFS_MAX_UNCOMPRESSED) {
> + if (!compress) {
> + if (em->len == BTRFS_MAX_UNCOMPRESSED)
> + ret = 0;
> + } else {
> + if (em->compress_type != compress_type)
> + goto out;
> + if (em->len == BTRFS_MAX_UNCOMPRESSED)
> + ret = 0;
> + }
> + }
> +
> out:
> /*
> * last_len ends up being a counter of how many bytes we've defragged.
> @@ -1477,7 +1498,8 @@ int btrfs_defrag_file(struct inode *inode, struct file
> *file,
>
> if (!should_defrag_range(inode, (u64)i << PAGE_SHIFT,
> extent_thresh, &last_len, &skip,
> -&defrag_end, do_compress)){
> +&defrag_end, do_compress,
> +compress_type)){
> unsigned long next;
> /*
> * the should_defrag function tells us how much to
> skip
> --
> 2.19.0
Ok, If no one like that patch,
may be at least fix autodefarag on compressed files?
By change default extent target size 256K -> 128K?
--
Have a nice day,
Timofey.
Re: [PATCH v2 3/9] geneirc/077 fix min size for btrfs
On Tue, Sep 25, 2018 at 12:24:16PM +0800, Anand Jain wrote: > If btrfs need to be tested at its default blockgroup which is non-mixed, > then it needs at least 256mb. > > Signed-off-by: Anand Jain (Sorry for the late review..) > --- > tests/generic/077 | 3 +-- > 1 file changed, 1 insertion(+), 2 deletions(-) > > diff --git a/tests/generic/077 b/tests/generic/077 > index ef6af18c83e3..ec236992513f 100755 > --- a/tests/generic/077 > +++ b/tests/generic/077 > @@ -49,8 +49,7 @@ rm -f $seqres.full > _scratch_unmount >/dev/null 2>&1 > echo "*** MKFS ***" >>$seqres.full > echo "" >>$seqres.full > -SIZE=`expr 50 \* 1024 \* 1024` > -_scratch_mkfs_sized $SIZE >>$seqres.full 2>&1 \ > +_scratch_mkfs_sized $((256 * 1024 *1024)) >>$seqres.full 2>&1 \ > || _fail "mkfs failed" Hmm, this test copies "/lib/modules/" to fill the original 50M filesystem (which seems a bad way to me) " # Something w/ enough data to fill 50M of fs... filler=/lib/modules/ ... echo "*** populate filesystem, pass #1" | tee -a $seqres.full cp -rf $filler $SCRATCH_MNT/subdir >$seqres.full 2>&1 ... " It works most of the time as "/lib/modules" is usually larger than 50M, but it may not fullfil the fs with 256M size. I think we should fix the way to fill the fs too. Thanks, Eryu > _scratch_mount > mkdir $SCRATCH_MNT/subdir > -- > 1.8.3.1 >
Re: [PATCH 09/15] vfs: pass operation flags to {clone, dedupe}_file_range implementations
On Fri, Oct 05, 2018 at 10:50:08AM -0700, Darrick J. Wong wrote: > > That's cool. But you know what's going to be the next step, right? > > Merging the 3 file operation interfaces into a single one. > > copy_file_range() already has the flags arg for future extensions > > and as you wrote somewhere, clone is really an optimized copy. > > ovl_copyfile() already does that internally. > > > > So the only take away for this patch series, please use constant > > names COPYRANGE_* and also explicitly define: > > > > /* Operation is actually clone, not copy. */ > > #define COPYRANGE_CLONE (1 << 2) > > /* Operation is actually dedupe, not copy. */ > > #define COPYRANGE_DEDUPE (1 << 3) > > Yeah, I was too tired to try to throw that one on top of the flaming > garbage pile. But I guess since I have a bunch more work to do to the > previous patch I might as well do that... I'm not totally sold on just merging everything, but I very much despise what is done in this patch, as it creates a completely confusing interface.
Re: [PATCH 08/15] vfs: change clone and dedupe range function pointers to return bytes completed
On Thu, Oct 04, 2018 at 05:45:35PM -0700, Darrick J. Wong wrote: > From: Darrick J. Wong > > Change the clone_file_range and dedupe_file_range functions to return > the number of bytes they operated on. This is the precursor to allowing > fs implementations to return short clone/dedupe results to the user, > which will enable us to obey resource limits in a graceful manner. For clone this doesn't make any sense, as we don't allow 'short clones'.
Re: [PATCH 07/15] vfs: skip zero-length dedupe requests
On Thu, Oct 04, 2018 at 05:45:21PM -0700, Darrick J. Wong wrote: > From: Darrick J. Wong > > Don't bother calling the filesystem for a zero-length dedupe request; > we can return zero and exit. > > Signed-off-by: Darrick J. Wong Looks fine, Reviewed-by: Christoph Hellwig
Re: [PATCH 05/15] vfs: check file ranges before cloning files
> +EXPORT_SYMBOL(generic_clone_checks); There doesn't seem to be any need to export this function. Otherwise looks good: Reviewed-by: Christoph Hellwig
Re: [PATCH 04/15] xfs: update ctime and remove suid before cloning files
> STATIC int
> xfs_reflink_remap_prep(
> @@ -1302,6 +1303,30 @@ xfs_reflink_remap_prep(
> /* Zap any page cache for the destination file's range. */
> truncate_inode_pages_range(&inode_out->i_data, pos_out,
> PAGE_ALIGN(pos_out + len) - 1);
> +
> + /* If we're altering the file contents... */
> + if (!is_dedupe) {
Nipick - even a clone might not alter the file content, we just have
no guarantee. So maybe change the comment to:
/* If we may alter the file contents.. */
Otherwise looks fine:
Reviewed-by: Christoph Hellwig
Re: [PATCH 03/15] xfs: zero posteof blocks when cloning above eof
On Thu, Oct 04, 2018 at 05:44:54PM -0700, Darrick J. Wong wrote: > From: Darrick J. Wong > > When we're reflinking between two files and the destination file range > is well beyond the destination file's EOF marker, zero any posteof > speculative preallocations in the destination file so that we don't > expose stale disk contents. The previous strategy of trying to clear > the preallocations does not work if the destination file has the > PREALLOC flag set. But I think we should still drop speculative delalloc preallocations instead of zeroing them in addition to zeroing of any real blocks in the data fork.
Re: [PATCH 02/15] xfs: refactor clonerange preparation into a separate helper
On Fri, Oct 05, 2018 at 03:28:09PM +1000, Dave Chinner wrote: > I've added: > > -- > This rework also moves the invalidation of the destination range to > the prep function so that it is done before the range is remapped. > This ensures that nobody can access the data in range being remapped > until the remap is complete. > -- > > Sound OK? > > Otherwise this looks fine. Given that the patch will need a respin anyway I'd rather split the move of the pagecache invalidation into a separate patch.
Re: [PATCH] fstests: btrfs verify hardening agaist duplicate fsid
On Mon, Oct 01, 2018 at 04:44:35PM +0800, Anand Jain wrote:
> We have a known bug in btrfs, that we let the device path be changed
> after the device has been mounted. So using this loop hole the new
> copied device would appears as if its mounted immediately after its
> been copied. So this test case reproduces this issue.
>
> For example:
>
> Initially.. /dev/mmcblk0p4 is mounted as /
>
> lsblk
> NAMEMAJ:MIN RM SIZE RO TYPE MOUNTPOINT
> mmcblk0 179:00 29.2G 0 disk
> |-mmcblk0p4 179:404G 0 part /
> |-mmcblk0p2 179:20 500M 0 part /boot
> |-mmcblk0p3 179:30 256M 0 part [SWAP]
> `-mmcblk0p1 179:10 256M 0 part /boot/efi
>
> btrfs fi show
> Label: none uuid: 07892354-ddaa-4443-90ea-f76a06accaba
> Total devices 1 FS bytes used 1.40GiB
> devid1 size 4.00GiB used 3.00GiB path /dev/mmcblk0p4
>
> Copy mmcblk0 to sda
> dd if=/dev/mmcblk0 of=/dev/sda
>
> And immediately after the copy completes the change in the device
> superblock is notified which the automount scans using
> btrfs device scan and the new device sda becomes the mounted root
> device.
>
> lsblk
> NAMEMAJ:MIN RM SIZE RO TYPE MOUNTPOINT
> sda 8:01 14.9G 0 disk
> |-sda48:414G 0 part /
> |-sda28:21 500M 0 part
> |-sda38:31 256M 0 part
> `-sda18:11 256M 0 part
> mmcblk0 179:00 29.2G 0 disk
> |-mmcblk0p4 179:404G 0 part
> |-mmcblk0p2 179:20 500M 0 part /boot
> |-mmcblk0p3 179:30 256M 0 part [SWAP]
> `-mmcblk0p1 179:10 256M 0 part /boot/efi
> btrfs fi show /
> Label: none uuid: 07892354-ddaa-4443-90ea-f76a06accaba
> Total devices 1 FS bytes used 1.40GiB
> devid1 size 4.00GiB used 3.00GiB path /dev/sda4
>
> The bug is quite nasty that you can't either unmount /dev/sda4 or
> /dev/mmcblk0p4. And the problem does not get solved until you take
> the sda out of the system on to another system to change its fsid using
> the 'btrfstune -u' command.
>
> Signed-off-by: Anand Jain
Looks like that the test will break the whole test env as it leaves an
unmountable $SCRATCH_MNT. I'd wait for the fix to get in first before
merging the test, in case it breaks normal regression tests. (I noticed
that the test is not in 'auto' group, so it's not that dangerous.)
Also, it'd be great if test can be reviewed by btrfs folks too!
> ---
> tests/btrfs/173 | 72
> +
> tests/btrfs/173.out | 5
> tests/btrfs/group | 1 +
> 3 files changed, 78 insertions(+)
> create mode 100755 tests/btrfs/173
> create mode 100644 tests/btrfs/173.out
>
> diff --git a/tests/btrfs/173 b/tests/btrfs/173
> new file mode 100755
> index ..f59a62e206c3
> --- /dev/null
> +++ b/tests/btrfs/173
> @@ -0,0 +1,72 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (c) 2018 Oracle. All Rights Reserved.
> +#
> +# FS QA Test 173
> +#
> +# Fuzzy test for FS image duplication.
> +# Could be fixed by
> +#[patch] btrfs: harden agaist duplicate fsid
> +#
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +# Modify as appropriate.
> +_supported_fs btrfs
> +_supported_os Linux
> +_require_scratch_dev_pool 2
> +_scratch_dev_pool_get 2
> +
> +dev_foo=$(echo $SCRATCH_DEV_POOL | awk '{print $1}' | rev | cut -d"/" -f1 |
> rev)
> +dev_bar=$(echo $SCRATCH_DEV_POOL | awk '{print $2}' | rev | cut -d"/" -f1 |
> rev)
This doesn't work if the devices in SCRATCH_DEV_POOL are symlinks, e.g.
lvm devices: /dev/mapper/testvg-testlv1, dev_foo is "testvg-testlv1" in
this case.
> +
> +_mkfs_dev /dev/$dev_foo
But /dev/testvg-testlv1 isn't existed.
_short_dev and/or _real_dev is useful in this case. e.g.
dev_foo=$(echo $SCRATCH_DEV_POOL | awk '{print $1}')
# dev_foo is like "dm-1"
dev_foo=$(_short_dev $dev_foo)
# dev_foo is like "/dev/dm-1"
dev_foo=$(_real_dev $dev_foo)
> +_mount /dev/$dev_foo $SCRATCH_MNT
It'd better to mount non-SCRATCH_DEV to other mount point, e.g.
$TEST_DIR/$seq.mnt
Thanks,
Eryu
> +
> +echo mount before btrfs image clone | tee -a $seqres.full
> +findmnt /dev/$dev_foo | grep -v TARGET | awk '{print $1" "$2}' | \
> + sed -e "s/$dev_foo/dev_foo/g" | _filter_scratch | tee -a $seqres.full
> +findmnt /dev/$dev_bar | grep -v TARGET | awk '{print $1" "$2}' | \
> + sed -e "s/$dev_bar/dev_bar/g" | _filter_scratch | tee -a $seqres.full
> +
> +for sb_bytenr in 65536 67108864
> +do
> + echo -n "dd status=none if=/dev/$dev_foo of=/dev/$dev_bar bs=1 "\
> + "seek=$sb_bytenr skip=$sb_bytenr c
Re: [PATCH v2 05/17] compat_ioctl: move more drivers to generic_compat_ioctl_ptrarg
On Wed 12 Sep 08:08 PDT 2018, Arnd Bergmann wrote:
[..]
> diff --git a/drivers/rpmsg/rpmsg_char.c b/drivers/rpmsg/rpmsg_char.c
> index a76b963a7e50..02aefb2b2d47 100644
> --- a/drivers/rpmsg/rpmsg_char.c
> +++ b/drivers/rpmsg/rpmsg_char.c
> @@ -285,7 +285,7 @@ static const struct file_operations rpmsg_eptdev_fops = {
> .write = rpmsg_eptdev_write,
> .poll = rpmsg_eptdev_poll,
> .unlocked_ioctl = rpmsg_eptdev_ioctl,
> - .compat_ioctl = rpmsg_eptdev_ioctl,
> + .compat_ioctl = generic_compat_ioctl_ptrarg,
> };
>
> static ssize_t name_show(struct device *dev, struct device_attribute *attr,
> @@ -446,7 +446,7 @@ static const struct file_operations rpmsg_ctrldev_fops = {
> .open = rpmsg_ctrldev_open,
> .release = rpmsg_ctrldev_release,
> .unlocked_ioctl = rpmsg_ctrldev_ioctl,
> - .compat_ioctl = rpmsg_ctrldev_ioctl,
> + .compat_ioctl = generic_compat_ioctl_ptrarg,
> };
>
For rpmsg part
Acked-by: Bjorn Andersson
Regards,
Bjorn
