Re: Input/output error on newly created file
Am Friday 13 May 2016 schrieb Duncan <1i5t5.dun...@cox.net> >Szalma László posted on Thu, 12 May 2016 20:28:24 +0200 as excerpted: >> The files that rarely become unreadable (I/O error but no error in dmesg >> or anywhere) are mysql MyIsam database files, and they are always small. >> Like 16kbyte for example, or smaller. Sometimes dropping the fs cache >> fixes the problem, sometimes not. Umount / mount always fixes the >> problem. Scrub says the filesystem is OK (when the file is unreadable). > >Is it possible the files are always under 4 KiB? For the record, I was seeing a similar error with dovecot *.index.log files (see the ML thread started by Szalma László) . In my case they are *not* all under 4 KiB. Looking at some of the affected files, one of them is 25K, and another is 6.6K. However, perhaps they compress to under 4K? But compressing the 25K one with lzop only goes down to 5.6K with -9 :-/ . >While there's a few variables as to max size, with 4 KiB being the >practical max, btrfs will inline really small files into their metadata >node instead of writing them out as 4 KiB data block extents. Since >that's obviously a different code-path, if it's only 4 KiB and smaller >files, it's possible there's a race in that code-path that when lost, >results in the file "disappearing" without a dmesg trace, only to >reappear after reboot. > >Actually, now that I think about it, if the files are all OVER say 2 KiB >but still quite small, say under 16 KiB, and if they had just recently >grown, it's possible that the race is in the transition from the inline >metadata state to the multiples of 4 KiB block data extent state. > >And if all the files had just shrunk, say from compaction (if done in- >place, not with a copy and rename), perhaps it's the reverse, the >transition from written data blocks to inline metadata state. I'm glad somebody is (publicly) thinking about this :-) ! Greetings -- Marc Joliet -- "People who think they know everything really annoy those of us who know we don't" - Bjarne Stroustrup signature.asc Description: This is a digitally signed message part.
Re: Input/output error on newly created file
Szalma László posted on Thu, 12 May 2016 20:28:24 +0200 as excerpted: > The files that rarely become unreadable (I/O error but no error in dmesg > or anywhere) are mysql MyIsam database files, and they are always small. > Like 16kbyte for example, or smaller. Sometimes dropping the fs cache > fixes the problem, sometimes not. Umount / mount always fixes the > problem. Scrub says the filesystem is OK (when the file is unreadable). Is it possible the files are always under 4 KiB? While there's a few variables as to max size, with 4 KiB being the practical max, btrfs will inline really small files into their metadata node instead of writing them out as 4 KiB data block extents. Since that's obviously a different code-path, if it's only 4 KiB and smaller files, it's possible there's a race in that code-path that when lost, results in the file "disappearing" without a dmesg trace, only to reappear after reboot. Actually, now that I think about it, if the files are all OVER say 2 KiB but still quite small, say under 16 KiB, and if they had just recently grown, it's possible that the race is in the transition from the inline metadata state to the multiples of 4 KiB block data extent state. And if all the files had just shrunk, say from compaction (if done in- place, not with a copy and rename), perhaps it's the reverse, the transition from written data blocks to inline metadata state. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Input/output error on newly created file
2016-05-12 19:41 keltezéssel, Nikolaus Rath írta: On May 12 2016, Diego Calleja wrote: El jueves, 12 de mayo de 2016 8:46:00 (CEST) Nikolaus Rath escribió: *ping* Anyone any idea? All I can say is that I've had the same problem in the past. In my case, the problematic files where active torrents. The interesting thing is that I was able to read them correctly up to a point, then I would get the same error as you. No messages in dmesg. The amount of data I was able to read from them was not random, it was something multiple of 4K. After reboot the problems went away and I wasn't able to reproduce it. There has been reports of similiar issues in the past: http://www.spinics.net/lists/linux-btrfs/msg52371.html Thanks for the pointer. So according to these reports, the IO errors happen with: 1. dm-crypt, LVM and mount options rw,noatime,compress=lzo,ssd,discard,space_cache,autodefrag,inode_cache 2. dm-crypt, no LVM, and mount options noatime,compress,nossd I can add the following data point: 3. dm-crypt on LVM, and mount options relatime,compress=lzo (Just to preserve this for the future) I don't want to repeat my e-mails in the past, but I think I have a similar issue. In short: xen virtual machine. The files that rarely become unreadable (I/O error but no error in dmesg or anywhere) are mysql MyIsam database files, and they are always small. Like 16kbyte for example, or smaller. Sometimes dropping the fs cache fixes the problem, sometimes not. Umount / mount always fixes the problem. Scrub says the filesystem is OK (when the file is unreadable). I experienced similar problem with log files (smtp or apache log files), but it is rare. It happens 1-2 times in a month on a heavy loaded mail/web server. (the log file io error is not a real problem for me, mysql is.). The problem present from 3.18 to 4.5. I started to migrate mysql table files to innodb, but i think this is only a workaround. László Szalma Best, -Nikolaus -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Input/output error on newly created file
On May 12 2016, Diego Calleja wrote: > El jueves, 12 de mayo de 2016 8:46:00 (CEST) Nikolaus Rath escribió: >> *ping* >> >> Anyone any idea? > > All I can say is that I've had the same problem in the past. In my > case, the problematic files where active torrents. The interesting > thing is that I was able to read them correctly up to a point, then > I would get the same error as you. No messages in dmesg. The amount > of data I was able to read from them was not random, it was > something multiple of 4K. After reboot the problems went away and > I wasn't able to reproduce it. > > There has been reports of similiar issues in the past: > http://www.spinics.net/lists/linux-btrfs/msg52371.html Thanks for the pointer. So according to these reports, the IO errors happen with: 1. dm-crypt, LVM and mount options rw,noatime,compress=lzo,ssd,discard,space_cache,autodefrag,inode_cache 2. dm-crypt, no LVM, and mount options noatime,compress,nossd I can add the following data point: 3. dm-crypt on LVM, and mount options relatime,compress=lzo (Just to preserve this for the future) Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Input/output error on newly created file
El jueves, 12 de mayo de 2016 8:46:00 (CEST) Nikolaus Rath escribió: > *ping* > > Anyone any idea? All I can say is that I've had the same problem in the past. In my case, the problematic files where active torrents. The interesting thing is that I was able to read them correctly up to a point, then I would get the same error as you. No messages in dmesg. The amount of data I was able to read from them was not random, it was something multiple of 4K. After reboot the problems went away and I wasn't able to reproduce it. There has been reports of similiar issues in the past: http://www.spinics.net/lists/linux-btrfs/msg52371.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Input/output error on newly created file
*ping* Anyone any idea? Best, -Nikolaus On May 09 2016, Nikolaus Rath wrote: > On May 09 2016, Filipe Manana wrote: >> On Sun, May 8, 2016 at 11:18 PM, Nikolaus Rath wrote: >>> Hello, >>> >>> I just created an innocent 10 MB on a btrfs file system, yet my attempt >>> to read it a few seconds later (and ever since), just gives: >>> >>> $ ls -l in-progress/mysterious-io-error >>> -rw-rw-r-- 1 nikratio nikratio 10485760 May 8 14:41 >>> in-progress/mysterious-io-error >>> $ cat in-progress/mysterious-io-error >>> cat: in-progress/mysterious-io-error: Input/output error >> >> If you unmount and mount again the filesystem, does it happen again? > > After rebooting, the previously unaccessible file can now be read. But I > cannot tell if it contains the right data. > > However, I just encountered the same problem with another, freshly > created file. > >> How did you create the file? > > In Python 3. The equivalent code is more or less: > > with open('file.dat', 'wb+') as fh: > for buf in generate_data(): > fh.write(buf) # bufsize is about 128 kB > > > However, I should note that there is a lot of activity in this > file system (it contains my home directory), so the above alone will > probably not reproduce the problem. > > That said, so far both the problematic files were created by the same > application (S3QL, of which luckily I am also the maintainer). > > >> Does fsck reports any issues? > > Do you mean btrfsck? It actually has a lot to say: > > checking extents > checking free space cache > checking fs roots > root 5 inode 3149867 errors 400, nbytes wrong > root 5 inode 3150237 errors 400, nbytes wrong > root 5 inode 3150238 errors 400, nbytes wrong > root 5 inode 3150242 errors 400, nbytes wrong > root 5 inode 3150260 errors 400, nbytes wrong > [...] > root 5 inode 15595011 errors 400, nbytes wrong > root 5 inode 15595016 errors 400, nbytes wrong > Checking filesystem on /dev/mapper/vg0-nikratio_crypt > UUID: 8742472d-a9b0-4ab6-b67a-5d21f14f7a38 > found 263648960636 bytes used err is 1 > total csum bytes: 395314372 > total tree bytes: 908644352 > total fs tree bytes: 352735232 > total extent tree bytes: 95039488 > btree space waste bytes: 156301160 > file data blocks allocated: 675209801728 > referenced 410351722496 > Btrfs v3.17 > > However, the inode of the problematic file (16186241) is not > mentioned. But I guess this is not surprising, because also for this > file, I can read the contents after remounting. > > > Best, > -Nikolaus > > > -- > GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F > Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F > > »Time flies like an arrow, fruit flies like a Banana.« > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Input/output error on newly created file
On May 09 2016, Filipe Manana wrote: > On Sun, May 8, 2016 at 11:18 PM, Nikolaus Rath wrote: >> Hello, >> >> I just created an innocent 10 MB on a btrfs file system, yet my attempt >> to read it a few seconds later (and ever since), just gives: >> >> $ ls -l in-progress/mysterious-io-error >> -rw-rw-r-- 1 nikratio nikratio 10485760 May 8 14:41 >> in-progress/mysterious-io-error >> $ cat in-progress/mysterious-io-error >> cat: in-progress/mysterious-io-error: Input/output error > > If you unmount and mount again the filesystem, does it happen again? After rebooting, the previously unaccessible file can now be read. But I cannot tell if it contains the right data. However, I just encountered the same problem with another, freshly created file. > How did you create the file? In Python 3. The equivalent code is more or less: with open('file.dat', 'wb+') as fh: for buf in generate_data(): fh.write(buf) # bufsize is about 128 kB However, I should note that there is a lot of activity in this file system (it contains my home directory), so the above alone will probably not reproduce the problem. That said, so far both the problematic files were created by the same application (S3QL, of which luckily I am also the maintainer). > Does fsck reports any issues? Do you mean btrfsck? It actually has a lot to say: checking extents checking free space cache checking fs roots root 5 inode 3149867 errors 400, nbytes wrong root 5 inode 3150237 errors 400, nbytes wrong root 5 inode 3150238 errors 400, nbytes wrong root 5 inode 3150242 errors 400, nbytes wrong root 5 inode 3150260 errors 400, nbytes wrong [...] root 5 inode 15595011 errors 400, nbytes wrong root 5 inode 15595016 errors 400, nbytes wrong Checking filesystem on /dev/mapper/vg0-nikratio_crypt UUID: 8742472d-a9b0-4ab6-b67a-5d21f14f7a38 found 263648960636 bytes used err is 1 total csum bytes: 395314372 total tree bytes: 908644352 total fs tree bytes: 352735232 total extent tree bytes: 95039488 btree space waste bytes: 156301160 file data blocks allocated: 675209801728 referenced 410351722496 Btrfs v3.17 However, the inode of the problematic file (16186241) is not mentioned. But I guess this is not surprising, because also for this file, I can read the contents after remounting. Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Input/output error on newly created file
On Sun, May 8, 2016 at 11:18 PM, Nikolaus Rath wrote: > Hello, > > I just created an innocent 10 MB on a btrfs file system, yet my attempt > to read it a few seconds later (and ever since), just gives: > > $ ls -l in-progress/mysterious-io-error > -rw-rw-r-- 1 nikratio nikratio 10485760 May 8 14:41 > in-progress/mysterious-io-error > $ cat in-progress/mysterious-io-error > cat: in-progress/mysterious-io-error: Input/output error If you unmount and mount again the filesystem, does it happen again? How did you create the file? Does fsck reports any issues? > > I was hoping the kernel might tell me a little more about this, but: > > $ dmesg | tail -3 > [ 48.982851] wlan0: RX AssocResp from 00:1f:33:be:9a:c4 (capab=0x411 > status=0 aid=3) > [ 48.983757] wlan0: associated > [ 2770.464505] kvm: zapping shadow pages for mmio generation wraparound > > > How can I find out the reason for the error? Is it a bug in BTRFS, > or somewhere else in the kernel ? Is my hardware broken? > > Kernel is Debian's 4.5.0-0.bpo.1-amd64. This is a btrfs file system on a > LVM LV backed by a bcache array with two backing devices and one caching > device. > > > Best, > -Nikolaus > > -- > GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F > Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F > > »Time flies like an arrow, fruit flies like a Banana.« > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Filipe David Manana, "Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men." -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html