Re: qcow2 images make scrub believe the filesystem is corrupted.

[Qu Wenruo]

On 2017年08月16日 10:28, Qu Wenruo wrote:


OK, data is not touched.
Single to single, so data chunks are not touched.
And your metadata is always good, so no problem should happen during 
balance.

Sorry, this part is wrong.

Data chunk is relocated, so I'm curious why there is no such kernel log 
warning about the csum mismatch.


Thanks,
Qu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: qcow2 images make scrub believe the filesystem is corrupted.

[Qu Wenruo]

On 2017年08月16日 09:51, Paulo Dias wrote:

Hi, thanks for the quick answer.

So, since i wrote this i tested this even further.

First, and as you predicted, if i try to cp the file to another
location i get read errors:

root@kerberos:/home/groo# cp Fedora/Fedora.qcow2 /
cp: error reading 'Fedora/Fedora.qcow2': Input/output error


Less possible to blame scrub now.
As normal read routine also reports such error, it maybe a real 
corruption of the file.




so i used this trick:

# modprobe nbd
# qemu-nbd --connect=/dev/nbd0 Fedora2.qcow2
# ddrescue /dev/nbd0 new_file.raw
# qemu-nbd --disconnect /dev/nbd0
# qemu-img convert -O qcow2 new_file.raw new_file.qcow2

and sure enough i was able to recreate the qcow2 but with this errors:

ago 15 22:19:49 kerberos kernel: block nbd0: Other side returned error (5)
ago 15 22:19:49 kerberos kernel: print_req_error: I/O error, dev nbd0,
sector 22159872
ago 15 22:19:49 kerberos kernel: BTRFS warning (device sda3): csum
failed root 258 ino 968837 off 17455849472 csum 0xcc028588 expected
csum 0xe3338de1 mirror 1


Still csum error.
And furthermore, both the expected and on-disk csum is not special value 
like crc32 for all zero page.

So it may means that, it's a real corruption.


ago 15 22:19:49 kerberos kernel: block nbd0: Other side returned error (5)
ago 15 22:19:49 kerberos kernel: print_req_error: I/O error, dev nbd0,
sector 22160016
ago 15 22:19:49 kerberos kernel: Buffer I/O error on dev nbd0, logical
block 2770002, async page read
ago 15 22:19:49 kerberos kernel: BTRFS warning (device sda3): csum
failed root 258 ino 968837 off 17455849472 csum 0xcc028588 expected
csum 0xe3338de1 mirror 1


At least, we now know which inode (968837 of root 258) and file offset 
(17455849472 length 4K) is corrupted.



ago 15 22:19:49 kerberos kernel: block nbd0: Other side returned error (5)
ago 15 22:19:49 kerberos kernel: print_req_error: I/O error, dev nbd0,
sector 22160016
ago 15 22:19:49 kerberos kernel: Buffer I/O error on dev nbd0, logical
block 2770002, async page read
ago 15 22:20:47 kerberos kernel: BTRFS warning (device sda3): csum
failed root 258 ino 968837 off 17455849472 csum 0xcc028588 expected
csum 0xe3338de1 mirror 1
ago 15 22:20:47 kerberos kernel: BTRFS warning (device sda3): csum
failed root 258 ino 968837 off 17455849472 csum 0xcc028588 expected
csum 0xe3338de1 mirror 1



block 2770002, async page read
ago 15 22:21:32 kerberos kernel: block nbd0: NBD_DISCONNECT
ago 15 22:21:32 kerberos kernel: block nbd0: shutting down sockets

i deleted the original Fedora.qcow2 and again scrub said i didnt had
any errors, so i wondered, could it be the raid1 code (long shot), so
i moved the metadata back to DUP.

btrfs fi balance start -dconvert=single -mconvert=dup /home/


OK, data is not touched.
Single to single, so data chunks are not touched.
And your metadata is always good, so no problem should happen during 
balance.


BTW, if you balance data, (no need to do convert, just balancing all 
data), it should also report error if my assumption is correct:

Some data is *really* corrupted.



root@kerberos:/home/groo# btrfs filesystem usage -T /home/
Overall:
 Device size: 333.50GiB
 Device allocated: 18.06GiB
 Device unallocated:  315.44GiB
 Device missing:  0.00B
 Used: 16.25GiB
 Free (estimated):315.83GiB  (min: 158.11GiB)
 Data ratio:   1.00
 Metadata ratio:   2.00
 Global reserve:   39.45MiB  (used: 0.00B)

  Data Metadata  System
Id Path  single   DUP   DUP  Unallocated
-- -  -  ---
  1 /dev/sda3 16.00GiB   2.00GiB 64.00MiB   181.94GiB
  2 /dev/sdb7- --   133.03GiB
  3 /dev/sdb8- --   488.13MiB
-- -  -  ---
Total 16.00GiB   1.00GiB 32.00MiB   315.44GiB
Used  15.61GiB 329.27MiB 16.00KiB

and once again copied the NEW fedora.qcow2 back to home and rerun scrub >
and once again i got errors:

root@kerberos:/home/groo# btrfs scrub start -B /home/
scrub done for ae9ae869-720d-4643-b673-6924d09b2fe0
 scrub started at Tue Aug 15 22:36:32 2017 and finished after 00:01:04
 total bytes scrubbed: 32.56GiB with 13 errors
 error details: csum=13
 corrected errors: 0, uncorrectable errors: 13, unverified errors: 0

ago 15 22:37:36 kerberos kernel: BTRFS error (device sda3): bdev
/dev/sda3 errs: wr 0, rd 0, flush 0, corrupt 35, gen 0
ago 15 22:37:36 kerberos kernel: BTRFS error (device sda3): unable to
fixup (regular) error at logical 418909777920 on dev /dev/sda3
ago 15 22:37:36 kerberos kernel: BTRFS error (device sda3): bdev



ago 15 22:37:36 kerberos kernel: BTRFS error (device sda3): bdev
/dev/sda3 errs: wr 0, rd 0, flush 0, corrupt 44, gen 0
ago 15 22:37:36 kerberos kernel: BTRFS error (device 

Re: qcow2 images make scrub believe the filesystem is corrupted.

[Paulo Dias]

Hi, thanks for the quick answer.

So, since i wrote this i tested this even further.

First, and as you predicted, if i try to cp the file to another
location i get read errors:

root@kerberos:/home/groo# cp Fedora/Fedora.qcow2 /
cp: error reading 'Fedora/Fedora.qcow2': Input/output error

so i used this trick:

# modprobe nbd
# qemu-nbd --connect=/dev/nbd0 Fedora2.qcow2
# ddrescue /dev/nbd0 new_file.raw
# qemu-nbd --disconnect /dev/nbd0
# qemu-img convert -O qcow2 new_file.raw new_file.qcow2

and sure enough i was able to recreate the qcow2 but with this errors:

ago 15 22:19:49 kerberos kernel: block nbd0: Other side returned error (5)
ago 15 22:19:49 kerberos kernel: print_req_error: I/O error, dev nbd0,
sector 22159872
ago 15 22:19:49 kerberos kernel: BTRFS warning (device sda3): csum
failed root 258 ino 968837 off 17455849472 csum 0xcc028588 expected
csum 0xe3338de1 mirror 1
ago 15 22:19:49 kerberos kernel: block nbd0: Other side returned error (5)
ago 15 22:19:49 kerberos kernel: print_req_error: I/O error, dev nbd0,
sector 22160016
ago 15 22:19:49 kerberos kernel: Buffer I/O error on dev nbd0, logical
block 2770002, async page read
ago 15 22:19:49 kerberos kernel: BTRFS warning (device sda3): csum
failed root 258 ino 968837 off 17455849472 csum 0xcc028588 expected
csum 0xe3338de1 mirror 1
ago 15 22:19:49 kerberos kernel: block nbd0: Other side returned error (5)
ago 15 22:19:49 kerberos kernel: print_req_error: I/O error, dev nbd0,
sector 22160016
ago 15 22:19:49 kerberos kernel: Buffer I/O error on dev nbd0, logical
block 2770002, async page read
ago 15 22:20:47 kerberos kernel: BTRFS warning (device sda3): csum
failed root 258 ino 968837 off 17455849472 csum 0xcc028588 expected
csum 0xe3338de1 mirror 1
ago 15 22:20:47 kerberos kernel: BTRFS warning (device sda3): csum
failed root 258 ino 968837 off 17455849472 csum 0xcc028588 expected
csum 0xe3338de1 mirror 1
ago 15 22:20:47 kerberos kernel: block nbd0: Other side returned error (5)
ago 15 22:20:47 kerberos kernel: print_req_error: I/O error, dev nbd0,
sector 22160016
ago 15 22:20:47 kerberos kernel: BTRFS warning (device sda3): csum
failed root 258 ino 968837 off 17455849472 csum 0xcc028588 expected
csum 0xe3338de1 mirror 1
ago 15 22:20:47 kerberos kernel: block nbd0: Other side returned error (5)
ago 15 22:20:47 kerberos kernel: print_req_error: I/O error, dev nbd0,
sector 22160016
ago 15 22:20:47 kerberos kernel: Buffer I/O error on dev nbd0, logical
block 2770002, async page read
ago 15 22:20:47 kerberos kernel: BTRFS warning (device sda3): csum
failed root 258 ino 968837 off 17455849472 csum 0xcc028588 expected
csum 0xe3338de1 mirror 1
ago 15 22:20:47 kerberos kernel: block nbd0: Other side returned error (5)
ago 15 22:20:47 kerberos kernel: print_req_error: I/O error, dev nbd0,
sector 22160016
ago 15 22:20:47 kerberos kernel: Buffer I/O error on dev nbd0, logical
block 2770002, async page read
ago 15 22:20:47 kerberos kernel: BTRFS warning (device sda3): csum
failed root 258 ino 968837 off 17455849472 csum 0xcc028588 expected
csum 0xe3338de1 mirror 1
ago 15 22:20:47 kerberos kernel: block nbd0: Other side returned error (5)
ago 15 22:20:47 kerberos kernel: print_req_error: I/O error, dev nbd0,
sector 22160016
ago 15 22:20:47 kerberos kernel: Buffer I/O error on dev nbd0, logical
block 2770002, async page read
ago 15 22:20:47 kerberos kernel: BTRFS warning (device sda3): csum
failed root 258 ino 968837 off 17455849472 csum 0xcc028588 expected
csum 0xe3338de1 mirror 1
ago 15 22:20:47 kerberos kernel: block nbd0: Other side returned error (5)
ago 15 22:20:47 kerberos kernel: print_req_error: I/O error, dev nbd0,
sector 22160016
ago 15 22:20:47 kerberos kernel: Buffer I/O error on dev nbd0, logical
block 2770002, async page read
ago 15 22:20:47 kerberos kernel: BTRFS warning (device sda3): csum
failed root 258 ino 968837 off 17455849472 csum 0xcc028588 expected
csum 0xe3338de1 mirror 1
ago 15 22:20:47 kerberos kernel: block nbd0: Other side returned error (5)
ago 15 22:20:47 kerberos kernel: print_req_error: I/O error, dev nbd0,
sector 22160016
ago 15 22:20:47 kerberos kernel: Buffer I/O error on dev nbd0, logical
block 2770002, async page read
ago 15 22:20:47 kerberos kernel: BTRFS warning (device sda3): csum
failed root 258 ino 968837 off 17455849472 csum 0xcc028588 expected
csum 0xe3338de1 mirror 1
ago 15 22:20:47 kerberos kernel: block nbd0: Other side returned error (5)
ago 15 22:20:47 kerberos kernel: print_req_error: I/O error, dev nbd0,
sector 22160016
ago 15 22:20:47 kerberos kernel: Buffer I/O error on dev nbd0, logical
block 2770002, async page read
ago 15 22:20:47 kerberos kernel: BTRFS warning (device sda3): csum
failed root 258 ino 968837 off 17455849472 csum 0xcc028588 expected
csum 0xe3338de1 mirror 1
ago 15 22:20:47 kerberos kernel: block nbd0: Other side returned error (5)
ago 15 22:20:47 kerberos kernel: print_req_error: I/O error, dev nbd0,
sector 22160016
ago 15 22:20:47 kerberos kernel: Buffer I/O error on dev nbd0, 

Re: qcow2 images make scrub believe the filesystem is corrupted.

[Qu Wenruo]

On 2017年08月16日 09:12, Paulo Dias wrote:

Hello/2 all

I'm using libvirt with a qcow2 image and everytime i run btrfs scrub
-H /home (subvolume where the image is), i get:

ago 15 21:58:08 kerberos kernel: BTRFS error (device sda3): bdev
/dev/sda3 errs: wr 0, rd 0, flush 0, corrupt 30, gen 0
ago 15 21:58:08 kerberos kernel: BTRFS error (device sda3): unable to
fixup (regular) error at logical 289831161856 on dev /dev/sda3
ago 15 21:58:08 kerberos kernel: BTRFS error (device sda3): bdev
/dev/sda3 errs: wr 0, rd 0, flush 0, corrupt 31, gen 0
ago 15 21:58:08 kerberos kernel: BTRFS error (device sda3): unable to
fixup (regular) error at logical 289830309888 on dev /dev/sda3
ago 15 21:58:08 kerberos kernel: BTRFS error (device sda3): bdev
/dev/sda3 errs: wr 0, rd 0, flush 0, corrupt 32, gen 0
ago 15 21:58:08 kerberos kernel: BTRFS error (device sda3): unable to
fixup (regular) error at logical 289831055360 on dev /dev/sda3
ago 15 21:58:08 kerberos kernel: BTRFS error (device sda3): bdev
/dev/sda3 errs: wr 0, rd 0, flush 0, corrupt 33, gen 0
ago 15 21:58:08 kerberos kernel: BTRFS error (device sda3): unable to
fixup (regular) error at logical 289861591040 on dev /dev/sda3
ago 15 21:58:09 kerberos kernel: BTRFS warning (device sda3): checksum
error at logical 290297204736 on dev /dev/sda3, sector 67982824, root
258, inode 968837, offset 17455849472, length 4096, links 1 (path:
groo/Fedora/Fedora.qcow2)


Any special setting on the file or the Fedora directory? Like nodatasum?

And is there any special setup like off-line dedupe?

Considering the number of corruption, only less than 50 and not 
continuous at all, it's a little weird.
For normal corruption, (at least on HDD) corruption range should be 
continuous, and more errors should be detected.



ago 15 21:58:09 kerberos kernel: BTRFS error (device sda3): bdev
/dev/sda3 errs: wr 0, rd 0, flush 0, corrupt 34, gen 0
ago 15 21:58:09 kerberos kernel: BTRFS error (device sda3): unable to
fixup (regular) error at logical 290297204736 on dev /dev/sda3

The thing is, as soon as i move the image to another subvolume, root
in this case, and delete it, the errors go away and scrub tells me i
have zero errors again.


This makes things even more weird.

If you're *moving* the file to another subvolume, its data still locates 
where it was, nothing is modified.


If you're *copying* the file to another subvolume, without reflinking, 
then kernel will try to read out the data and write it back to new place.
During the read, it will verify data checksum. And if it doesn't match, 
you'll get EIO error during the copy.


If you're *reflinking* the file, using cp --reflink=always, it's the 
same result as *moving*.


Anyway, the data of your image is either kept as it is, or re-written to 
new place.
If there is really some corruption, for copy case you should get some 
error, and for moving/reflinking case, scrub will always report error.


I doubt if there is something wrong with scrub.

Can you even reproduce it with a smaller sparse file? For example 
several mega size.

And is it only happening in that specified Fedora directory?

Thanks,
Qu



Then if i AGAIN copy the file back to /home, i get the same errors.

qemu-img check tells me the qcow2 file is fine, and smart doesnt show
me anything wrong with my ssd:

root@kerberos:/home/groo# smartctl -Ai /dev/sda
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.13.0-041300rc4-generic]
(local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Samsung based SSDs
Device Model: Samsung SSD 850 EVO M.2 500GB
Serial Number:S33DNX0H812686V
LU WWN Device Id: 5 002538 d4130d027
Firmware Version: EMT21B6Q
User Capacity:500.107.862.016 bytes [500 GB]
Sector Size:  512 bytes logical/physical
Rotation Rate:Solid State Device
Form Factor:  M.2
Device is:In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ATA8-ACS T13/1699-D revision 4c
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:Tue Aug 15 21:59:34 2017 -03
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME  FLAG VALUE WORST THRESH TYPE
UPDATED  WHEN_FAILED RAW_VALUE
   5 Reallocated_Sector_Ct   0x0033   100   100   010Pre-fail
Always   -   0
   9 Power_On_Hours  0x0032   099   099   000Old_age
Always   -   1739
  12 Power_Cycle_Count   0x0032   099   099   000Old_age
Always   -   392
177 Wear_Leveling_Count 0x0013   099   099   000Pre-fail
Always   -   7
179 Used_Rsvd_Blk_Cnt_Tot   0x0013   100   100   010Pre-fail
Always   -   0
181 Program_Fail_Cnt_Total  0x0032   100   100   010Old_age
Always   -   0
182 Erase_Fail_Count_Total  

qcow2 images make scrub believe the filesystem is corrupted.

[Paulo Dias]

Hello/2 all

I'm using libvirt with a qcow2 image and everytime i run btrfs scrub
-H /home (subvolume where the image is), i get:

ago 15 21:58:08 kerberos kernel: BTRFS error (device sda3): bdev
/dev/sda3 errs: wr 0, rd 0, flush 0, corrupt 30, gen 0
ago 15 21:58:08 kerberos kernel: BTRFS error (device sda3): unable to
fixup (regular) error at logical 289831161856 on dev /dev/sda3
ago 15 21:58:08 kerberos kernel: BTRFS error (device sda3): bdev
/dev/sda3 errs: wr 0, rd 0, flush 0, corrupt 31, gen 0
ago 15 21:58:08 kerberos kernel: BTRFS error (device sda3): unable to
fixup (regular) error at logical 289830309888 on dev /dev/sda3
ago 15 21:58:08 kerberos kernel: BTRFS error (device sda3): bdev
/dev/sda3 errs: wr 0, rd 0, flush 0, corrupt 32, gen 0
ago 15 21:58:08 kerberos kernel: BTRFS error (device sda3): unable to
fixup (regular) error at logical 289831055360 on dev /dev/sda3
ago 15 21:58:08 kerberos kernel: BTRFS error (device sda3): bdev
/dev/sda3 errs: wr 0, rd 0, flush 0, corrupt 33, gen 0
ago 15 21:58:08 kerberos kernel: BTRFS error (device sda3): unable to
fixup (regular) error at logical 289861591040 on dev /dev/sda3
ago 15 21:58:09 kerberos kernel: BTRFS warning (device sda3): checksum
error at logical 290297204736 on dev /dev/sda3, sector 67982824, root
258, inode 968837, offset 17455849472, length 4096, links 1 (path:
groo/Fedora/Fedora.qcow2)
ago 15 21:58:09 kerberos kernel: BTRFS error (device sda3): bdev
/dev/sda3 errs: wr 0, rd 0, flush 0, corrupt 34, gen 0
ago 15 21:58:09 kerberos kernel: BTRFS error (device sda3): unable to
fixup (regular) error at logical 290297204736 on dev /dev/sda3

The thing is, as soon as i move the image to another subvolume, root
in this case, and delete it, the errors go away and scrub tells me i
have zero errors again.

Then if i AGAIN copy the file back to /home, i get the same errors.

qemu-img check tells me the qcow2 file is fine, and smart doesnt show
me anything wrong with my ssd:

root@kerberos:/home/groo# smartctl -Ai /dev/sda
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.13.0-041300rc4-generic]
(local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Samsung based SSDs
Device Model: Samsung SSD 850 EVO M.2 500GB
Serial Number:S33DNX0H812686V
LU WWN Device Id: 5 002538 d4130d027
Firmware Version: EMT21B6Q
User Capacity:500.107.862.016 bytes [500 GB]
Sector Size:  512 bytes logical/physical
Rotation Rate:Solid State Device
Form Factor:  M.2
Device is:In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ATA8-ACS T13/1699-D revision 4c
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:Tue Aug 15 21:59:34 2017 -03
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME  FLAG VALUE WORST THRESH TYPE
UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   100   100   010Pre-fail
Always   -   0
  9 Power_On_Hours  0x0032   099   099   000Old_age
Always   -   1739
 12 Power_Cycle_Count   0x0032   099   099   000Old_age
Always   -   392
177 Wear_Leveling_Count 0x0013   099   099   000Pre-fail
Always   -   7
179 Used_Rsvd_Blk_Cnt_Tot   0x0013   100   100   010Pre-fail
Always   -   0
181 Program_Fail_Cnt_Total  0x0032   100   100   010Old_age
Always   -   0
182 Erase_Fail_Count_Total  0x0032   100   100   010Old_age
Always   -   0
183 Runtime_Bad_Block   0x0013   100   100   010Pre-fail
Always   -   0
187 Uncorrectable_Error_Cnt 0x0032   100   100   000Old_age
Always   -   0
190 Airflow_Temperature_Cel 0x0032   061   050   000Old_age
Always   -   39
195 ECC_Error_Rate  0x001a   200   200   000Old_age
Always   -   0
199 CRC_Error_Count 0x003e   100   100   000Old_age
Always   -   0
235 POR_Recovery_Count  0x0012   099   099   000Old_age
Always   -   54
241 Total_LBAs_Written  0x0032   099   099   000Old_age
Always   -   7997549567

this is the usage for /home:

root@kerberos:/home/groo# btrfs filesystem usage -T /home/
Overall:
Device size: 333.50GiB
Device allocated: 74.12GiB
Device unallocated:  259.38GiB
Device missing:  0.00B
Used: 32.70GiB
Free (estimated):297.36GiB  (min: 167.67GiB)
Data ratio:   1.00
Metadata ratio:   2.00
Global reserve:   58.12MiB  (used: 0.00B)

 Data Metadata  System
Id Path  single   RAID1 RAID1Unallocated
-- -  -  

Re: [PATCH] btrfs-progs: fix cross-compile build

[Qu Wenruo]

On 2017年08月16日 02:11, Eric Sandeen wrote:

The mktables binary needs to be build with the host
compiler at built time, not the target compiler, because
it runs at build time to generate the raid tables.

Copy auto-fu from xfsprogs and modify Makefile to
accomodate this.

Reported-by: Hallo32 
Signed-off-by: Eric Sandeen 


Looks better than my previous patch.
With @BUILD_CLFAGS support and better BUILD_CC/CLFAGS detection for 
native build environment.


Reviewed-by: Qu Wenruo 

Thanks,
Qu

---

diff --git a/Makefile b/Makefile
index b3e2b63..2647b95 100644
--- a/Makefile
+++ b/Makefile
@@ -323,7 +323,7 @@ version.h: version.sh version.h.in configure.ac
  
  mktables: kernel-lib/mktables.c

@echo "[CC] $@"
-   $(Q)$(CC) $(CFLAGS) $< -o $@
+   $(Q)$(BUILD_CC) $(BUILD_CFLAGS) $< -o $@
  
  kernel-lib/tables.c: mktables

@echo "[TABLE]  $@"
diff --git a/Makefile.inc.in b/Makefile.inc.in
index 4e1b68c..0570bf8 100644
--- a/Makefile.inc.in
+++ b/Makefile.inc.in
@@ -4,6 +4,8 @@
  export
  
  CC = @CC@

+BUILD_CC = @BUILD_CC@
+BUILD_CFLAGS = @BUILD_CFLAGS@
  LN_S = @LN_S@
  AR = @AR@
  RM = @RM@
diff --git a/configure.ac b/configure.ac
index 30055f8..bc590cc 100644
--- a/configure.ac
+++ b/configure.ac
@@ -26,6 +26,23 @@ AC_CONFIG_SRCDIR([btrfs.c])
  AC_PREFIX_DEFAULT([/usr/local])
  
  AC_PROG_CC

+AC_ARG_VAR(BUILD_CC, [C compiler for build tools])
+if test "${BUILD_CC+set}" != "set"; then
+  if test $cross_compiling = no; then
+BUILD_CC="$CC"
+  else
+AC_CHECK_PROGS(BUILD_CC, gcc cc) > +  fi
+fi
+AC_ARG_VAR(BUILD_CFLAGS, [C compiler flags for build tools])
+if test "${BUILD_CFLAGS+set}" != "set"; then
+  if test $cross_compiling = no; then
+BUILD_CFLAGS="$CFLAGS"
+  else
+BUILD_CFLAGS="-g -O2"
+  fi
+fi
+
  AC_CANONICAL_HOST
  AC_C_CONST
  AC_C_VOLATILE

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs-progs-v4.12: cross compiling

[Qu Wenruo]

On 2017年08月16日 01:28, Eric Sandeen wrote:

On 8/15/17 7:48 AM, David Sterba wrote:

On Tue, Aug 15, 2017 at 02:44:07PM +0200, Hallo32 wrote:


...


How the kernel deals with this kind of problem ?
Looking at the source of btrfs Makefile, it is more simple to replace

mktables: kernel-lib/mktables.c
  @echo "[CC] $@"
  $(Q)$(CC) $(CFLAGS) $< -o $@

with


mktables: kernel-lib/mktables.c
  @echo "[HOSTCC] $@"
  $(Q)$(HOSTCC) $(CFLAGS) $< -o $@

where HOSTCC is defined as

HOSTCC=gcc


(may be the same applied also to CFLAGS <-> HOSTCFLAGS ?)


If using HOSTCC then I'm fine with it.


CFLAGS needs also be replaced by something like HOSTCFLAGS, because if
you use something like mips/architecture specific CFLAGS, they may be
not applicably on the host system.


Good point. Without a regular/automated cross-compilation build testing
I think we could break it quite easily. I'm going to keep the
pregenerated file in git.


Isn't using the host compiler for some binaries during a cross-compile
a very standard thing to do?

The kernel manages it, as shown above.  xfsprogs does it (see libxfs/Makefile
for the crc32table.h and crc32selftest targets), e2fsprogs does it (see
gen_crc32ctable target in lib/ext2fs/Makefile), etc.


Only if I can find it earlier.



Seems that checking in a generated file would be more prone to eventual
errors, no?


Also my concern.



I guess it's harder to do in btrfs-progs since it's not using autotools...


And your patch does do a better job, especially for BUILD_CFLAGS and 
build cc detection.


Thanks,
Qu



-Eric


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 4/4 v3] btrfs: add compression trace points

[kbuild test robot]

Hi Anand,

[auto build test WARNING on tip/perf/core]
[also build test WARNING on v4.13-rc5]
[cannot apply to btrfs/next next-20170815]
[if your patch is applied to the wrong git tree, please drop us a note to help 
improve the system]

url:
https://github.com/0day-ci/linux/commits/Anand-Jain/misc-compression-tracing-related-patches/20170816-043401
config: s390-allmodconfig (attached as .config)
compiler: s390x-linux-gnu-gcc (Debian 6.1.1-9) 6.1.1 20160705
reproduce:
wget 
https://raw.githubusercontent.com/01org/lkp-tests/master/sbin/make.cross -O 
~/bin/make.cross
chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
make.cross ARCH=s390 

All warnings (new ones prefixed by >>):

   In file included from include/trace/define_trace.h:95:0,
from include/trace/events/btrfs.h:1674,
from fs/btrfs/super.c:65:
   include/trace/events/btrfs.h: In function 'trace_raw_output_btrfs_compress':
>> include/trace/events/btrfs.h:91:12: warning: format '%lu' expects argument 
>> of type 'long unsigned int', but argument 6 has type 'ino_t {aka unsigned 
>> int}' [-Wformat=]
 TP_printk("%pU: " fmt, __entry->fsid, args)
   ^
   include/trace/trace_events.h:359:22: note: in definition of macro 
'DECLARE_EVENT_CLASS'
 trace_seq_printf(s, print); \
 ^
   include/trace/trace_events.h:78:9: note: in expansion of macro 'PARAMS'
PARAMS(print)); \
^~
>> include/trace/events/btrfs.h:1632:1: note: in expansion of macro 
>> 'TRACE_EVENT'
TRACE_EVENT(btrfs_compress,
^~~
>> include/trace/events/btrfs.h:91:2: note: in expansion of macro 'TP_printk'
 TP_printk("%pU: " fmt, __entry->fsid, args)
 ^
>> include/trace/events/btrfs.h:1664:2: note: in expansion of macro 
>> 'TP_printk_btrfs'
 TP_printk_btrfs("%s %s ino=%lu type=%s len_before=%lu len_after=%lu 
start=%lu ret=%d",
 ^~~

vim +91 include/trace/events/btrfs.h

bc074524 Jeff Mahoney 2016-06-09  78  
bc074524 Jeff Mahoney 2016-06-09  79  #define TP_fast_assign_fsid(fs_info)  
\
bc074524 Jeff Mahoney 2016-06-09  80memcpy(__entry->fsid, fs_info->fsid, 
BTRFS_UUID_SIZE)
bc074524 Jeff Mahoney 2016-06-09  81  
bc074524 Jeff Mahoney 2016-06-09  82  #define TP_STRUCT__entry_btrfs(args...)   
\
bc074524 Jeff Mahoney 2016-06-09  83TP_STRUCT__entry(   
\
bc074524 Jeff Mahoney 2016-06-09  84TP_STRUCT__entry_fsid   
\
bc074524 Jeff Mahoney 2016-06-09  85args)
bc074524 Jeff Mahoney 2016-06-09  86  #define TP_fast_assign_btrfs(fs_info, 
args...)\
bc074524 Jeff Mahoney 2016-06-09  87TP_fast_assign( 
\
bc074524 Jeff Mahoney 2016-06-09  88TP_fast_assign_fsid(fs_info);   
\
bc074524 Jeff Mahoney 2016-06-09  89args)
bc074524 Jeff Mahoney 2016-06-09  90  #define TP_printk_btrfs(fmt, args...) \
bc074524 Jeff Mahoney 2016-06-09 @91TP_printk("%pU: " fmt, __entry->fsid, 
args)
8c2a3ca2 Josef Bacik  2012-01-10  92  

:: The code at line 91 was first introduced by commit
:: bc074524e123ded281cde25ebc5661910f9679e3 btrfs: prefix fsid to all trace 
events

:: TO: Jeff Mahoney <je...@suse.com>
:: CC: David Sterba <dste...@suse.com>

---
0-DAY kernel test infrastructureOpen Source Technology Center
https://lists.01.org/pipermail/kbuild-all   Intel Corporation


.config.gz
Description: application/gzip

Re: btrfs-progs-v4.12: cross compiling

[Jeff Mahoney]

On 8/14/17 11:10 AM, David Sterba wrote:
> On Mon, Aug 14, 2017 at 10:14:42PM +0800, Qu Wenruo wrote:
>> On 2017年08月14日 22:03, David Sterba wrote:
>>> On Mon, Aug 14, 2017 at 09:17:08PM +0800, Qu Wenruo wrote:
 On 2017年08月14日 21:06, David Sterba wrote:
> On Mon, Aug 14, 2017 at 02:17:26PM +0200, Hallo32 wrote:
>> Since versions 4.12 btrfs-progs is complicated to cross compile for
>> other systems.
>> The problem is, that this version includes mktables, which needs to be
>> compiled for the host system and executed there for the creation of
>> tables.c.
>>
>> Are there any changes planed for the next version of btrfs-progs to make
>> the cross compiling as simple as in the past? A included tables.c for
>> example?
>
> Yes, keeping the generated tables.c around is fine. There's no reason it
> needs to be generated each time during build. I'll fix that in 4.12.1.

 But the number of lines and impossibility to review it makes it not
 suitable to be managed by git.
>>>
>>> I don't understand your concern. The file is generated from a set of
>>> formulas, not intended to be updated directly.
>>
>> Yes, it should never be updated directly, so it's generated by a less 
>> than 400 lines program, instead of a whole 10K+ lines file managed by git.
> 
> mktables.c is synced from kernel sources, taking updates from there is
> easier than porting any changes to the proposed scripted implementation.
> 
> The workflow is simple:
> - copy kernel mktables.c changes to btrfs-progs mktables.c
> - compile mktables
> - run 'make kernel-lib/tables.c'

Can't this happen as part of a make dist (that we don't do right now)?

> - commit the changes to git

... and anyone using the git repo directly can sort out how to build it?

-Jeff

-- 
Jeff Mahoney
SUSE Labs



signature.asc
Description: OpenPGP digital signature

Re: btrfs-progs-v4.12: cross compiling

[Eric Sandeen]

On 8/15/17 12:28 PM, Eric Sandeen wrote:
> I guess it's harder to do in btrfs-progs since it's not using autotools...

Eh, I don't know why I thought that was still true :)  patch sent.

-Eric
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH] btrfs-progs: fix cross-compile build

[Eric Sandeen]

The mktables binary needs to be build with the host
compiler at built time, not the target compiler, because
it runs at build time to generate the raid tables.

Copy auto-fu from xfsprogs and modify Makefile to
accomodate this.

Reported-by: Hallo32 
Signed-off-by: Eric Sandeen 
---

diff --git a/Makefile b/Makefile
index b3e2b63..2647b95 100644
--- a/Makefile
+++ b/Makefile
@@ -323,7 +323,7 @@ version.h: version.sh version.h.in configure.ac
 
 mktables: kernel-lib/mktables.c
@echo "[CC] $@"
-   $(Q)$(CC) $(CFLAGS) $< -o $@
+   $(Q)$(BUILD_CC) $(BUILD_CFLAGS) $< -o $@
 
 kernel-lib/tables.c: mktables
@echo "[TABLE]  $@"
diff --git a/Makefile.inc.in b/Makefile.inc.in
index 4e1b68c..0570bf8 100644
--- a/Makefile.inc.in
+++ b/Makefile.inc.in
@@ -4,6 +4,8 @@
 export
 
 CC = @CC@
+BUILD_CC = @BUILD_CC@
+BUILD_CFLAGS = @BUILD_CFLAGS@
 LN_S = @LN_S@
 AR = @AR@
 RM = @RM@
diff --git a/configure.ac b/configure.ac
index 30055f8..bc590cc 100644
--- a/configure.ac
+++ b/configure.ac
@@ -26,6 +26,23 @@ AC_CONFIG_SRCDIR([btrfs.c])
 AC_PREFIX_DEFAULT([/usr/local])
 
 AC_PROG_CC
+AC_ARG_VAR(BUILD_CC, [C compiler for build tools])
+if test "${BUILD_CC+set}" != "set"; then
+  if test $cross_compiling = no; then
+BUILD_CC="$CC"
+  else
+AC_CHECK_PROGS(BUILD_CC, gcc cc)
+  fi
+fi
+AC_ARG_VAR(BUILD_CFLAGS, [C compiler flags for build tools])
+if test "${BUILD_CFLAGS+set}" != "set"; then
+  if test $cross_compiling = no; then
+BUILD_CFLAGS="$CFLAGS"
+  else
+BUILD_CFLAGS="-g -O2"
+  fi
+fi
+
 AC_CANONICAL_HOST
 AC_C_CONST
 AC_C_VOLATILE

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] Btrfs: incremental send, fix emission of invalid clone operations

[Liu Bo]

On Thu, Aug 10, 2017 at 10:54:51PM +0100, fdman...@kernel.org wrote:
> From: Filipe Manana 
> 
> When doing an incremental send it's possible that the computed send stream
> contains clone operations that will fail on the receiver if the receiver
> has compression enabled and the clone operations target a sector sized
> extent that starts at a zero file offset, is not compressed on the source
> filesystem but ends up being compressed and inlined at the destination
> filesystem.
> 
> Example scenario:
> 
>   $ mkfs.btrfs -f /dev/sdb
>   $ mount -o compress /dev/sdb /mnt
> 
>   # By doing a direct IO write, the data is not compressed.
>   $ xfs_io -f -d -c "pwrite -S 0xab 0 4K" /mnt/foobar
>   $ btrfs subvolume snapshot -r /mnt /mnt/mysnap1
> 
>   $ xfs_io -c "reflink /mnt/foobar 0 8K 4K" /mnt/foobar
>   $ btrfs subvolume snapshot -r /mnt /mnt/mysnap2
> 
>   $ btrfs send -f /tmp/1.snap /mnt/mysnap1
>   $ btrfs send -f /tmp/2.snap -p /mnt/mysnap1 /mnt/mysnap2
>   $ umount /mnt
> 
>   $ mkfs.btrfs -f /dev/sdc
>   $ mount -o compress /dev/sdc /mnt
>   $ btrfs receive -f /tmp/1.snap /mnt
>   $ btrfs receive -f /tmp/2.snap /mnt
>   ERROR: failed to clone extents to foobar
>   Operation not supported
> 
> The same could be achieved by mounting the source filesystem without
> compression and doing a buffered IO write instead of a direct IO one,
> and mounting the destination filesystem with compression enabled.
> 
> So fix this by issuing regular write operations in the send stream
> instead of clone operations when the source offset is zero and the
> range has a length matching the sector size.

Reviewed-by: Liu Bo 

Thanks,

-liubo
> 
> Signed-off-by: Filipe Manana 
> ---
>  fs/btrfs/send.c | 19 +++
>  1 file changed, 19 insertions(+)
> 
> diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c
> index b082210df9c8..460be72ab78b 100644
> --- a/fs/btrfs/send.c
> +++ b/fs/btrfs/send.c
> @@ -4992,6 +4992,25 @@ static int clone_range(struct send_ctx *sctx,
>   struct btrfs_key key;
>   int ret;
>  
> + /*
> +  * Prevent cloning from a zero offset with a length matching the sector
> +  * size because in some scenarios this will make the receiver fail.
> +  *
> +  * For example, if in the source filesystem the extent at offset 0
> +  * has a length of sectorsize and it was written using direct IO, then
> +  * it can never be an inline extent (even if compression is enabled).
> +  * Then this extent can be cloned in the original filesystem to a non
> +  * zero file offset, but it may not be possible to clone in the
> +  * destination filesystem because it can be inlined due to compression
> +  * on the destination filesystem (as the receiver's write operations are
> +  * always done using buffered IO). The same happens when the original
> +  * filesystem does not have compression enabled but the destination
> +  * filesystem has.
> +  */
> + if (clone_root->offset == 0 &&
> + len == sctx->send_root->fs_info->sectorsize)
> + return send_extent_data(sctx, offset, len);
> +
>   path = alloc_path_for_send();
>   if (!path)
>   return -ENOMEM;
> -- 
> 2.11.0
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs-progs-v4.12: cross compiling

[Eric Sandeen]

On 8/15/17 7:48 AM, David Sterba wrote:
> On Tue, Aug 15, 2017 at 02:44:07PM +0200, Hallo32 wrote:

...

 How the kernel deals with this kind of problem ?
 Looking at the source of btrfs Makefile, it is more simple to replace

mktables: kernel-lib/mktables.c
  @echo "[CC] $@"
  $(Q)$(CC) $(CFLAGS) $< -o $@

 with


mktables: kernel-lib/mktables.c
  @echo "[HOSTCC] $@"
  $(Q)$(HOSTCC) $(CFLAGS) $< -o $@

 where HOSTCC is defined as

 HOSTCC=gcc


 (may be the same applied also to CFLAGS <-> HOSTCFLAGS ?)
>>>
>>> If using HOSTCC then I'm fine with it.
>>
>> CFLAGS needs also be replaced by something like HOSTCFLAGS, because if 
>> you use something like mips/architecture specific CFLAGS, they may be 
>> not applicably on the host system.
> 
> Good point. Without a regular/automated cross-compilation build testing
> I think we could break it quite easily. I'm going to keep the
> pregenerated file in git.

Isn't using the host compiler for some binaries during a cross-compile
a very standard thing to do?

The kernel manages it, as shown above.  xfsprogs does it (see libxfs/Makefile
for the crc32table.h and crc32selftest targets), e2fsprogs does it (see
gen_crc32ctable target in lib/ext2fs/Makefile), etc.

Seems that checking in a generated file would be more prone to eventual
errors, no?

I guess it's harder to do in btrfs-progs since it's not using autotools...

-Eric
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RedHat 7.4 Release Notes: "Btrfs has been deprecated" - wut?

[Austin S. Hemmelgarn]

On 2017-08-15 10:41, Christoph Anton Mitterer wrote:

On Tue, 2017-08-15 at 07:37 -0400, Austin S. Hemmelgarn wrote:

Go look at Chrome, or Firefox, or Opera, or any other major web
browser.
   At minimum, they will safely bail out if they detect corruption in
the
user profile and can trivially resync the profile from another system
if
the user has profile sync set up.


Aha,... I'd rather see a concrete reference to some white paper or
code, where one can really see that these programs actually *do* their
own checksumming.
But even from what you claim here now (that they'd only detect the
corruption and then resync from another system - which is nothing else
than recovering from a backup), I wouldn't see the big problem with
EIO.
It isn't a problem if it isn't a false positive.  It is a problem when 
it's not correct and the data is accurate.  This breaks from current 
behavior on BTRFS in a not insignificant way.  As things stand right 
now, -EIO on BTRFS means one of two things:

* The underlying device returned an IO error.
* The data there is incorrect.

While it technically is possible for there to be a false positive with 
CoW, it is a statistical impossibility even at Google and Facebook scale 
(I will comment that I've had this happen (exactly once), but it 
resulted from severe widespread media issues in the storage device that 
should have caused catastrophic failure of the device).


There is no way to avoid false positives without CoW or journaling.  We 
have CoW, and people aren't using it for performance reasons.  Adding 
journaling instead will make performance worse (and brings up the 
important question of whether or not the journal is CoW) for NOCOW, and 
has the potential to make performance worse than without NOCOW.




Go take a look at any enterprise
database application from a reasonable company, it will almost
always
support replication across systems and validate data it reads.


Okay, I already showed you, that PostgreSQL, MySQL, BDB, sqlite can't
or don't do per default... so which do you mean with the enterprise DB
(Oracle?) and where's the reference that shows that they really do
general checksuming? And that EIO would be a problem for their recovery
strategies?
Again, I never said it had to be checksumming.  Type and range checking 
and validation of the metadata (not through checksumming, but through 
verifying that the metadata makes sense, essentially the equivalent of 
fsck on older filesystems) _is_ done by almost everything dealing with 
databases these days except for trivial one-off stuff.


As far as EIO, see my reply above.


And again, we're not talking about the WALs (or whatever these programs
call it) which are there to handle a crash... we are talking about
silent data corruption.
Reread what I said.  Database _APPLICATION_ is not the same as database 
system.  PGSQL, MySQL, BDB, SQLite, MSSQL, Oracle, etc, are all database 
systems, they provide a database that an application can build on top 
of, and yes, none of them provide any significant protection (except 
possibly MSSQL, but I'm not sure about that and it's not hugely relevant 
to this particular discussion).  Things like MythTV, Bugzilla, Kodi, and 
other stuff that utilize the database for back-end storage (including 
things like many media players and web browsers) are database 
applications.  The distinction here is no different from Linux 
applications versus Linux systems.


In the context of actual applications using the database, it's still not 
rigorous verification like you seem to think I'm talking about, but most 
of them do enough sanity checking that most stuff beyond single bit 
errors in numeric and string types will be caught and at least reported.>



Agreed, but there's also the counter argument for log files that
most
people who are not running servers rarely (if ever) look at old
logs,
and it's the old logs that are the most likely to have at rest
corruption (the longer something sits idle on media, the more likely
it
will suffer from a media error).


I wouldn't have any valid prove that it's really the "idle" data, which
is the most likely one to have silent corruptions (at least not for all
types of storage medium), but even if this is the case as you say...
then it's probably more likely to hit the /usr/ /lib/ and so on stuff
on stable distros... logs are typically rotated and then at least once
re-written (when compressed).
Except that /usr and /lib are trivial to validate on any modern Linux or 
BSD system because the package manager almost certainly has file 
validation built in.  At minimum, emerge, Entropy, DNF, yum, FreeBSD 
pkg-ng, pkgin, Zypper, YaST2, Nix, and Alpine APK, all have this 
functionality, and there is at least one readily available piece of 
software (debsigs) for dpkg based systems.  Sensibly security minded 
individuals generally already have this type of validation in a cron job 
or systemd timer.




Go install OpenSUSE in a VM.  Look at what 

Re: RedHat 7.4 Release Notes: "Btrfs has been deprecated" - wut?

[Christoph Anton Mitterer]

On Tue, 2017-08-15 at 07:37 -0400, Austin S. Hemmelgarn wrote:
> Go look at Chrome, or Firefox, or Opera, or any other major web
> browser. 
>   At minimum, they will safely bail out if they detect corruption in
> the 
> user profile and can trivially resync the profile from another system
> if 
> the user has profile sync set up.

Aha,... I'd rather see a concrete reference to some white paper or
code, where one can really see that these programs actually *do* their
own checksumming.
But even from what you claim here now (that they'd only detect the
corruption and then resync from another system - which is nothing else
than recovering from a backup), I wouldn't see the big problem with
EIO.


> Go take a look at any enterprise 
> database application from a reasonable company, it will almost
> always 
> support replication across systems and validate data it reads.

Okay, I already showed you, that PostgreSQL, MySQL, BDB, sqlite can't
or don't do per default... so which do you mean with the enterprise DB
(Oracle?) and where's the reference that shows that they really do
general checksuming? And that EIO would be a problem for their recovery
strategies?

And again, we're not talking about the WALs (or whatever these programs
call it) which are there to handle a crash... we are talking about
silent data corruption.



> Agreed, but there's also the counter argument for log files that
> most 
> people who are not running servers rarely (if ever) look at old
> logs, 
> and it's the old logs that are the most likely to have at rest 
> corruption (the longer something sits idle on media, the more likely
> it 
> will suffer from a media error).

I wouldn't have any valid prove that it's really the "idle" data, which
is the most likely one to have silent corruptions (at least not for all
types of storage medium), but even if this is the case as you say...
then it's probably more likely to hit the /usr/ /lib/ and so on stuff
on stable distros... logs are typically rotated and then at least once
re-written (when compressed).


> Go install OpenSUSE in a VM.  Look at what filesystem it uses.  Go 
> install Solaris in a VM, lo and behold it uses ZFS _with no option
> for 
> anything else_ as it's root filesystem.  Go install a recent version
> of 
> Windows server in a VM, notice that it also has the option of a
> properly 
> checked filesystem (ReFS).  Go install FreeBSD in a VM, notice that
> it 
> provides the option (which is actively recommended by many people
> who 
> use FreeBSD) to install with root on ZFS.  Install Android or Chrome
> OS 
> (or AOSP or Chromium OS) in a VM.  Root the system and take a look
> at 
> the storage stack, both of them use dm-verity, and Android (and
> possibly 
> Chrome OS too, not 100% certain) uses per-file AEAD through the VFS 
> encryption API on encrypted devices.

So your argument for not adding support for this is basically:
People don't or shouldn't use btrfs for this? o.O



>   The fact that some OS'es blindly 
> trust the underlying storage hardware is not our issue, it's their 
> issue, and it shouldn't be 'fixed' by BTRFS because it doesn't just 
> affect their customers who run the OS in a VM on BTRFS.

Then you can probably drop checksumming from btrfs altogether. And with
the same "argument" any other advanced feature.
For resilience there is hardware RAID or Linux' MD raid... so no need
to keep it in btrfs o.O


> Most enterprise database apps offer support for
> replication, 
> and quite a few do their own data validation when reading from the 
> database.
First of all,... replication != the capability to detect silent data
corruption.

You still haven't named a single one which does checksumming per
default. At least those which are quite popular in the FLOSS world all
don't seem to do.



Cheers,
Chris.

smime.p7s
Description: S/MIME cryptographic signature

Re: [PATCH 1/3] copied android.mk from devel branch

[David Sterba]

On Wed, Aug 02, 2017 at 11:51:09AM -0700, filipbystri...@google.com wrote:
> From: Filip Bystricky 
> 
> This series of patches fixes some compile errors that trigger when
> compiling to android devices.
> 
> This first patch just brings in devel's Android.mk, to which
> kdave@ added a few fixes recently.
> 
> Signed-off-by: Filip Bystricky 
> Reviewed-by: Mark Salyzyn 

The changes in this patch are already applied in current master or devel
branch, so the resulting diff is empty. The patches 2 and 3 bring the
real fixes so I'm going to apply them but patch 2 lacks the changelog.
It should briefly describe the type of changes.

As you've probably fixed the current compilation issues, I'd like to
avoid future breakage when we update the standard Makefile. The travis
CI could be used in connection with the docker images, but I'm open to
other suggestions to do verify builds in other CI systems if there's
some already established practice.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs-progs-v4.12: cross compiling

[David Sterba]

On Tue, Aug 15, 2017 at 02:44:07PM +0200, Hallo32 wrote:
> Am 15.08.2017 um 01:39 schrieb Qu Wenruo:
> > On 2017年08月15日 02:57, Goffredo Baroncelli wrote:
> >> On 08/14/2017 05:10 PM, David Sterba wrote:
> >>> On Mon, Aug 14, 2017 at 10:14:42PM +0800, Qu Wenruo wrote:
> >> [...]
> >>> mktables.c is synced from kernel sources, taking updates from there is
> >>> easier than porting any changes to the proposed scripted 
> >>> implementation.
> >>>
> >>> The workflow is simple:
> >>> - copy kernel mktables.c changes to btrfs-progs mktables.c
> >>
> >> How the kernel deals with this kind of problem ?
> >> Looking at the source of btrfs Makefile, it is more simple to replace
> >>
> >>mktables: kernel-lib/mktables.c
> >>  @echo "[CC] $@"
> >>  $(Q)$(CC) $(CFLAGS) $< -o $@
> >>
> >> with
> >>
> >>
> >>mktables: kernel-lib/mktables.c
> >>  @echo "[HOSTCC] $@"
> >>  $(Q)$(HOSTCC) $(CFLAGS) $< -o $@
> >>
> >> where HOSTCC is defined as
> >>
> >> HOSTCC=gcc
> >>
> >>
> >> (may be the same applied also to CFLAGS <-> HOSTCFLAGS ?)
> >
> > If using HOSTCC then I'm fine with it.
> 
> CFLAGS needs also be replaced by something like HOSTCFLAGS, because if 
> you use something like mips/architecture specific CFLAGS, they may be 
> not applicably on the host system.

Good point. Without a regular/automated cross-compilation build testing
I think we could break it quite easily. I'm going to keep the
pregenerated file in git.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs-progs-v4.12: cross compiling

[Hallo32]

Am 15.08.2017 um 01:39 schrieb Qu Wenruo:



On 2017年08月15日 02:57, Goffredo Baroncelli wrote:

On 08/14/2017 05:10 PM, David Sterba wrote:

On Mon, Aug 14, 2017 at 10:14:42PM +0800, Qu Wenruo wrote:

[...]

mktables.c is synced from kernel sources, taking updates from there is
easier than porting any changes to the proposed scripted 
implementation.


The workflow is simple:
- copy kernel mktables.c changes to btrfs-progs mktables.c


How the kernel deals with this kind of problem ?
Looking at the source of btrfs Makefile, it is more simple to replace

   mktables: kernel-lib/mktables.c
 @echo "[CC] $@"
 $(Q)$(CC) $(CFLAGS) $< -o $@

with


   mktables: kernel-lib/mktables.c
 @echo "[HOSTCC] $@"
 $(Q)$(HOSTCC) $(CFLAGS) $< -o $@

where HOSTCC is defined as

HOSTCC=gcc


(may be the same applied also to CFLAGS <-> HOSTCFLAGS ?)


If using HOSTCC then I'm fine with it.

Thanks,
Qu


Hi,

CFLAGS needs also be replaced by something like HOSTCFLAGS, because if 
you use something like mips/architecture specific CFLAGS, they may be 
not applicably on the host system.


Best regards





- compile mktables
- run 'make kernel-lib/tables.c'
- commit the changes to git

All of that happens very rarely, if ever, the raid56 tables and
correction algorithms are set in stone. Any extensions need to be done
on both sides kernel/userspace.


What about using script to generate it?


We do have the mktables utility to generate it and I'll regenerate it
should there be a change to kernel-lib/mktables.c


I mean to replace mktables.c with a script.
So no cross compiler problems at all, and even easier Makefile.
No dependence on "mktables" program.


Somebody has to implement the script and verify that the output is the
same, eventually sync changes. The cross-compilation should be fixed
with the pregenerated tables.c . Is Makefile size really a concern? The
number of related lines is like 7. I don't see any benefit in what you
propose and hopefully explained my viewpoint enough so I don't have to
continue.
--
To unsubscribe from this list: send the line "unsubscribe 
linux-btrfs" in

the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html








--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RedHat 7.4 Release Notes: "Btrfs has been deprecated" - wut?

[Austin S. Hemmelgarn]

On 2017-08-14 15:54, Christoph Anton Mitterer wrote:

On Mon, 2017-08-14 at 11:53 -0400, Austin S. Hemmelgarn wrote:

Quite a few applications actually _do_ have some degree of secondary
verification or protection from a crash.  Go look at almost any
database
software.

Then please give proper references for this!

This is from 2015, where you claimed this already and I looked up all
the bigger DBs and they either couldn't do it at all, didn't to it per
default, or it required application support (i.e. from the programs
using the DB)
https://www.spinics.net/lists/linux-btrfs/msg50258.html
Go look at Chrome, or Firefox, or Opera, or any other major web browser. 
 At minimum, they will safely bail out if they detect corruption in the 
user profile and can trivially resync the profile from another system if 
the user has profile sync set up.  Go take a look at any enterprise 
database application from a reasonable company, it will almost always 
support replication across systems and validate data it reads.  Note 
that in both cases this isn't the same as BTRFS checking block 
checksums, and I never said that the application had to work without 
issue, even BTRFS and ZFS can only provide that guarantee with multiple 
devices or dup profiles on a single disk, but I can count on one hand 
the software I've used in the last few years that didn't at least fail 
gracefully when fed bad data (and sending -EIO when a checksum fails is 
essentially the same thing).



It usually will not have checksumming, but it will almost
always have support for a journal, which is enough to cover the
particular data loss scenario we're talking about (unexpected
unclean
shutdown).


I don't think we talk about this:
We talk about people wanting checksuming to notice e.g. silent data
corruption.

The crash case is only the corner case about what happens then if data
is written correctly but csums not.



In my own experience, the things that use nodatacow fall into one of
4
categories:
1. Cases where the data is non-critical, and data loss will be
inconvenient but not fatal.  Systemd journal files are a good example
of
this, as are web browser profiles when you're using profile sync.


I'd guess many people would want to have their log files valid and
complete. Same for their profiles (especially since people concerned
about their integrity might not want to have these synced to
Mozilla/Google etc.)
Agreed, but there's also the counter argument for log files that most 
people who are not running servers rarely (if ever) look at old logs, 
and it's the old logs that are the most likely to have at rest 
corruption (the longer something sits idle on media, the more likely it 
will suffer from a media error).




2. Cases where the upper level can reasonably be expected to have
some
degree of handling, even if it's not correction.  VM disk images and
most database applications fall into this category.


No. Wrong. Or prove me that I'm wrong ;-)
And these two (VMs, DBs) are actually *the* main cases for nodatacow.
Go install OpenSUSE in a VM.  Look at what filesystem it uses.  Go 
install Solaris in a VM, lo and behold it uses ZFS _with no option for 
anything else_ as it's root filesystem.  Go install a recent version of 
Windows server in a VM, notice that it also has the option of a properly 
checked filesystem (ReFS).  Go install FreeBSD in a VM, notice that it 
provides the option (which is actively recommended by many people who 
use FreeBSD) to install with root on ZFS.  Install Android or Chrome OS 
(or AOSP or Chromium OS) in a VM.  Root the system and take a look at 
the storage stack, both of them use dm-verity, and Android (and possibly 
Chrome OS too, not 100% certain) uses per-file AEAD through the VFS 
encryption API on encrypted devices.  The fact that some OS'es blindly 
trust the underlying storage hardware is not our issue, it's their 
issue, and it shouldn't be 'fixed' by BTRFS because it doesn't just 
affect their customers who run the OS in a VM on BTRFS.


As far as databases, I know of only one piece of enterprise level 
database software that doesn't have some kind of handling for this type 
of thing, and it's a a horribly designed piece of software other than 
that too.  Most enterprise database apps offer support for replication, 
and quite a few do their own data validation when reading from the 
database.  And if you care about non-enterprise database apps, then you 
need to worry about the edge case caused by unclean shutdown.




And I and most other sysadmins I know would prefer the opposite with
the
addition of a secondary notification method.  You can still hook the
notification to stop the application, but you don't have to if you
don't
want to (and in cases 1 and 2 I listed above, you probably don't want
to).


Then I guess btrfs is generally not the right thing for such people, as
in the CoW case it will also give them EIO on any corruptions and their
programs will fail.
For a single disk?  Yes, 

Re: [PATCH] fstests: btrfs: enhance regression test for nocsum dio read's repair

[Lu Fengqi]

On Tue, Aug 15, 2017 at 05:16:06PM +0800, Eryu Guan wrote:
>On Mon, Aug 14, 2017 at 03:03:13PM +0800, Lu Fengqi wrote:
>> I catch this following error from dmesg when this testcase fails.
>> 
>> [17446.661127] Buffer I/O error on dev sdb1, logical block 64, async page 
>> read
>> 
>> We expect to inject disk IO errors on the device when xfs_io reads the
>> specific file, but other processes may trigger IO error earlier. So, we
>> can use task-filter to solve this problem.
>> 
>> Signed-off-by: Lu Fengqi 
>
>This looks OK to me. Does btrfs/143 need a similar fix?
>
>Thanks,
>Eryu
>
>

Although btrfs/143 has a similar problem, this method doesn't work for it.
Unlike btrfs/142, the second IO error needs to be triggered by another process,
not by xfs_io. So we can't simply set task-filter to solve this problem.

I'm still investigating a way to solve the problem.

Do you have any ideas?
Any help will be greatly appreciated.

-- 
Thanks,
Lu


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] fstests: btrfs: enhance regression test for nocsum dio read's repair

[Eryu Guan]

On Mon, Aug 14, 2017 at 03:03:13PM +0800, Lu Fengqi wrote:
> I catch this following error from dmesg when this testcase fails.
> 
> [17446.661127] Buffer I/O error on dev sdb1, logical block 64, async page read
> 
> We expect to inject disk IO errors on the device when xfs_io reads the
> specific file, but other processes may trigger IO error earlier. So, we
> can use task-filter to solve this problem.
> 
> Signed-off-by: Lu Fengqi 

This looks OK to me. Does btrfs/143 need a similar fix?

Thanks,
Eryu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] btrfs-progs: Fix cross-compile error for mktables

[Qu Wenruo]

On 2017年08月15日 15:14, Goffredo Baroncelli wrote:

Hi Qu,

On 08/15/2017 04:19 AM, Qu Wenruo wrote:

When cross compiling btrfs-progs, following error will prevent
btrfs-progs to be compiled:

 [CC] mktables
 [TABLE]  kernel-lib/tables.c
/bin/sh: ./mktables: cannot execute binary file: Exec format error
make: *** No rule to make target 'kernel-lib/tables.c', needed by 
'kernel-lib/tables.o'.  Stop.

"mktables" should only be executed in host environment, while @CC
set by autoconf will follow host/build/target setting, causing mktables
to be cross-compiled.

The fix is to introduce a new @HOSTCC for mktables, which will not be
affected by host/build/target settings.

Reported-by: Hallo32 
Suggested-by: David Sterba 


this idea was a my suggestion :(


Sorry, I though all that 2 mails are from David.
And David still likes the idea to let git manage that file...

So correct tag should be:

Suggested-by: Goffredo Baroncelli 

Thanks for pointing this out.
Qu



Signed-off-by: Qu Wenruo 
---
Tested with AArch64 cross-toolchain created by buildroot.
---
  Makefile| 2 +-
  Makefile.inc.in | 1 +
  configure.ac| 1 +
  3 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index b3e2b636..0395e37f 100644
--- a/Makefile
+++ b/Makefile
@@ -323,7 +323,7 @@ version.h: version.sh version.h.in configure.ac
  
  mktables: kernel-lib/mktables.c

@echo "[CC] $@"
-   $(Q)$(CC) $(CFLAGS) $< -o $@
+   $(Q)$(HOSTCC) $(CFLAGS) $< -o $@
  
  kernel-lib/tables.c: mktables

@echo "[TABLE]  $@"
diff --git a/Makefile.inc.in b/Makefile.inc.in
index 4e1b68cb..308acca3 100644
--- a/Makefile.inc.in
+++ b/Makefile.inc.in
@@ -4,6 +4,7 @@
  export
  
  CC = @CC@

+HOSTCC = @HOSTCC@
  LN_S = @LN_S@
  AR = @AR@
  RM = @RM@
diff --git a/configure.ac b/configure.ac
index 30055f85..f6051ebd 100644
--- a/configure.ac
+++ b/configure.ac
@@ -26,6 +26,7 @@ AC_CONFIG_SRCDIR([btrfs.c])
  AC_PREFIX_DEFAULT([/usr/local])
  
  AC_PROG_CC

+AC_PATH_PROGS([HOSTCC], [gcc clang])
  AC_CANONICAL_HOST
  AC_C_CONST
  AC_C_VOLATILE





--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH] btrfs: remove redundant check on ret being non-zero

[Colin King]

From: Colin Ian King 

The error return variable ret is initialized to zero and then is
checked to see if it is non-zero in the if-block that follows it.
It is therefore impossible for ret to be non-zero after the if-block
hence the check is redundant and can be removed.

Detected by CoverityScan, CID#1021040 ("Logically dead code")

Signed-off-by: Colin Ian King 
---
 fs/btrfs/tree-log.c | 2 --
 1 file changed, 2 deletions(-)

diff --git a/fs/btrfs/tree-log.c b/fs/btrfs/tree-log.c
index 3a11ae63676e..f05fcc67efa6 100644
--- a/fs/btrfs/tree-log.c
+++ b/fs/btrfs/tree-log.c
@@ -1143,8 +1143,6 @@ static inline int __add_inode_ref(struct 
btrfs_trans_handle *trans,
goto again;
}
kfree(victim_name);
-   if (ret)
-   return ret;
 next:
cur_offset += victim_name_len + sizeof(*extref);
}
-- 
2.11.0

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html