Re: fio reports data corruption with btrfs
On Mon, Jun 25, 2012 at 12:30:34PM -0600, Alex Lyakas wrote: > Greetings everybody, > > I am running a fio test on btrfs compiled from > git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs.git, > up to commit: > cb77fcd88569cd2b7b25ecd4086ea932a53be9b3 Btrfs: delay iput with async extents > including this commit. > > Below is a fio configuration file, and later fio textual output. > Here: > https://docs.google.com/folder/d/0B1AuaIB8xZtbNTRuSW1zVGozWFE/edit > are "expected" vs "received" mismatch reports. Strangely, when I read > the mismatched block from the file reported as corrupted by fio, I > receive data different both from "expected" and "received" blocks that > fio reports. I added one such file (job0.1.0.88576.now) to the > pastebin as well. > > If you think that my fio configuration file is faulty, please let me > know. fio version is 1.59. The idea is to run 10 io processes in > parallel. > Mount options? I'm running the test now, I'll let you know if I can reproduce. Thanks, Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: fio reports data corruption with btrfs
On Mon, Jun 25, 2012 at 12:30:34PM -0600, Alex Lyakas wrote: > Greetings everybody, > > I am running a fio test on btrfs compiled from > git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs.git, > up to commit: > cb77fcd88569cd2b7b25ecd4086ea932a53be9b3 Btrfs: delay iput with async extents > including this commit. > > Below is a fio configuration file, and later fio textual output. > Here: > https://docs.google.com/folder/d/0B1AuaIB8xZtbNTRuSW1zVGozWFE/edit > are "expected" vs "received" mismatch reports. Strangely, when I read > the mismatched block from the file reported as corrupted by fio, I > receive data different both from "expected" and "received" blocks that > fio reports. I added one such file (job0.1.0.88576.now) to the > pastebin as well. > > If you think that my fio configuration file is faulty, please let me > know. fio version is 1.59. The idea is to run 10 io processes in > parallel. > So we think it may be enospc, so try btrfs-next git://git.kernel.org/pub/scm/linux/kernel/git/josef/btrfs-next.git which has an enospc fix related to creating a crapptone of files. Let me know if that helps. Thanks, Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: fio reports data corruption with btrfs
Hi Josef, Mount options were noatime, nodatacow. So you say that fio might have received ENOSPC, but didn't abort the test? I will compile your branch and let you know. I did not see any error messages from the kernel, except from: Jun 25 10:04:28 vc kernel: [ 436.730890] btrfs: setting nodatacow Jun 25 10:04:28 vc kernel: [ 436.744139] btrfs: no dev_stats entry found for device /dev/sdb2 (devid 1) (OK on first mount after mkfs) Jun 25 10:13:12 vc kernel: [ 960.844149] INFO: task flush-btrfs-2:3349 blocked for more than 120 seconds. Jun 25 10:13:12 vc kernel: [ 960.846600] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jun 25 10:13:12 vc kernel: [ 960.847507] flush-btrfs-2 D 8180ca80 0 3349 2 0x Jun 25 10:13:12 vc kernel: [ 960.847515] 8801186337a0 0046 13e332ba 81c1d780 Jun 25 10:13:12 vc kernel: [ 960.847520] 880118633fd8 880118633fd8 880118633fd8 00013840 Jun 25 10:13:12 vc kernel: [ 960.847525] 81c13020 8801176f5b80 880118633790 88011fc140e8 Jun 25 10:13:12 vc kernel: [ 960.847530] Call Trace: Jun 25 10:13:12 vc kernel: [ 960.847554] [] schedule+0x29/0x70 Jun 25 10:13:12 vc kernel: [ 960.847558] [] io_schedule+0x8f/0xd0 Jun 25 10:13:12 vc kernel: [ 960.847574] [] get_request_wait+0xef/0x240 Jun 25 10:13:12 vc kernel: [ 960.847587] [] ? add_wait_queue+0x60/0x60 Jun 25 10:13:12 vc kernel: [ 960.847592] [] blk_queue_bio+0x7f/0x3a0 Jun 25 10:13:12 vc kernel: [ 960.847596] [] generic_make_request.part.50+0x74/0xb0 Jun 25 10:13:12 vc kernel: [ 960.847600] [] generic_make_request+0x68/0x70 Jun 25 10:13:12 vc kernel: [ 960.847603] [] submit_bio+0x87/0x110 Jun 25 10:13:12 vc kernel: [ 960.847649] [] btrfs_map_bio+0x167/0x210 [btrfs] Jun 25 10:13:12 vc kernel: [ 960.847669] [] btrfs_submit_bio_hook+0x7d/0x140 [btrfs] Jun 25 10:13:12 vc kernel: [ 960.847691] [] submit_one_bio+0x6a/0xa0 [btrfs] Jun 25 10:13:12 vc kernel: [ 960.847713] [] flush_epd_write_bio+0x39/0x50 [btrfs] Jun 25 10:13:12 vc kernel: [ 960.847734] [] extent_writepages+0x50/0x60 [btrfs] Jun 25 10:13:12 vc kernel: [ 960.847754] [] ? btrfs_submit_direct+0x1e0/0x1e0 [btrfs] Jun 25 10:13:12 vc kernel: [ 960.847759] [] ? bit_waitqueue+0x14/0xc0 Jun 25 10:13:12 vc kernel: [ 960.847779] [] btrfs_writepages+0x28/0x30 [btrfs] Jun 25 10:13:12 vc kernel: [ 960.847793] [] do_writepages+0x21/0x40 Jun 25 10:13:12 vc kernel: [ 960.847805] [] writeback_single_inode+0x112/0x380 Jun 25 10:13:12 vc kernel: [ 960.847809] [] writeback_sb_inodes+0x1b6/0x270 Jun 25 10:13:12 vc kernel: [ 960.847813] [] __writeback_inodes_wb+0x9e/0xd0 Jun 25 10:13:12 vc kernel: [ 960.847816] [] wb_writeback+0x28b/0x340 Jun 25 10:13:12 vc kernel: [ 960.847823] [] ? __switch_to+0x137/0x410 Jun 25 10:13:12 vc kernel: [ 960.847833] [] ? get_nr_dirty_inodes+0x52/0x80 Jun 25 10:13:12 vc kernel: [ 960.847837] [] wb_check_old_data_flush+0x9f/0xb0 Jun 25 10:13:12 vc kernel: [ 960.847842] [] wb_do_writeback+0x149/0x1d0 Jun 25 10:13:12 vc kernel: [ 960.847848] [] ? usleep_range+0x50/0x50 Jun 25 10:13:12 vc kernel: [ 960.847852] [] bdi_writeback_thread+0x8b/0x290 Jun 25 10:13:12 vc kernel: [ 960.847855] [] ? wb_do_writeback+0x1d0/0x1d0 Jun 25 10:13:12 vc kernel: [ 960.847860] [] kthread+0x93/0xa0 Jun 25 10:13:12 vc kernel: [ 960.847868] [] kernel_thread_helper+0x4/0x10 Jun 25 10:13:12 vc kernel: [ 960.847873] [] ? kthread_freezable_should_stop+0x70/0x70 Jun 25 10:13:12 vc kernel: [ 960.847877] [] ? gs_change+0x13/0x13 Thanks, Alex. On Mon, Jun 25, 2012 at 10:26 PM, Josef Bacik wrote: > On Mon, Jun 25, 2012 at 12:30:34PM -0600, Alex Lyakas wrote: >> Greetings everybody, >> >> I am running a fio test on btrfs compiled from >> git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs.git, >> up to commit: >> cb77fcd88569cd2b7b25ecd4086ea932a53be9b3 Btrfs: delay iput with async extents >> including this commit. >> >> Below is a fio configuration file, and later fio textual output. >> Here: >> https://docs.google.com/folder/d/0B1AuaIB8xZtbNTRuSW1zVGozWFE/edit >> are "expected" vs "received" mismatch reports. Strangely, when I read >> the mismatched block from the file reported as corrupted by fio, I >> receive data different both from "expected" and "received" blocks that >> fio reports. I added one such file (job0.1.0.88576.now) to the >> pastebin as well. >> >> If you think that my fio configuration file is faulty, please let me >> know. fio version is 1.59. The idea is to run 10 io processes in >> parallel. >> > > So we think it may be enospc, so try btrfs-next > > git://git.kernel.org/pub/scm/linux/kernel/git/josef/btrfs-next.git > > which has an enospc fix related to creating a crapptone of files. Let me know > if that helps. Thanks, > > Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo in
Re: fio reports data corruption with btrfs
Hi Josef, I have rerun the test with btrfs-next master branch. fio reported mismatched blocks again. Mount options were the same (-o noatime,nodatacow). In both cases the drive is a 135Gb drive, while the total size of allocated block groups is around 60Gb: Data: total=62.01GB, used=49.04GB System, DUP: total=8.00MB, used=12.00KB System: total=4.00MB, used=0.00 Metadata, DUP: total=3.00GB, used=5.58MB Metadata: total=8.00MB, used=0.00 (In addition, one of the fio processes crashed with the following stack trace: Program terminated with signal 6, Aborted. #0 0x7fbd66ddf445 in raise () from /lib/x86_64-linux-gnu/libc.so.6 (gdb) bt #0 0x7fbd66ddf445 in raise () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x7fbd66de2bab in abort () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x7fbd66e1ce2e in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #3 0x7fbd66e27626 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #4 0x00419e91 in verify_io_u () #5 0x0041cc0e in ?? () #6 0x0041cf49 in io_u_sync_complete () #7 0x0040b090 in ?? () #8 0x0040b4ae in ?? () #9 0x004076a3 in main () I will recompile fio with symbols for later tests). I have put all the files here (including core), just in case: https://docs.google.com/folder/d/0B1AuaIB8xZtbb3ExRk5qRVFjYWc/edit fio output: job0: (g=0): rw=randrw, bs=512-64K/512-64K, ioengine=sync, iodepth=1 job1: (g=0): rw=randrw, bs=512-64K/512-64K, ioengine=sync, iodepth=1 job2: (g=0): rw=randrw, bs=512-64K/512-64K, ioengine=sync, iodepth=1 job3: (g=0): rw=randrw, bs=512-64K/512-64K, ioengine=sync, iodepth=1 job4: (g=0): rw=randrw, bs=512-64K/512-64K, ioengine=sync, iodepth=1 job5: (g=0): rw=randrw, bs=512-64K/512-64K, ioengine=sync, iodepth=1 job6: (g=0): rw=randrw, bs=512-64K/512-64K, ioengine=sync, iodepth=1 job7: (g=0): rw=randrw, bs=512-64K/512-64K, ioengine=sync, iodepth=1 job8: (g=0): rw=randrw, bs=512-64K/512-64K, ioengine=sync, iodepth=1 job9: (g=0): rw=randrw, bs=512-64K/512-64K, ioengine=sync, iodepth=1 fio 1.59 Starting 10 processes job0: Laying out IO file(s) (1000 file(s) / 4997MB) job1: Laying out IO file(s) (1000 file(s) / 4921MB) job2: Laying out IO file(s) (1000 file(s) / 5140MB) job3: Laying out IO file(s) (1000 file(s) / 5086MB) job4: Laying out IO file(s) (1000 file(s) / 4869MB) job5: Laying out IO file(s) (1000 file(s) / 5106MB) job6: Laying out IO file(s) (1000 file(s) / 4980MB) job7: Laying out IO file(s) (1000 file(s) / 5052MB) job8: Laying out IO file(s) (1000 file(s) / 5075MB) job9: Laying out IO file(s) (1000 file(s) / 4964MB) md5: verify failed at file /mnt/btrfs/job5.6.0 offset 9728, length 512 Expected CRC: Received CRC: e3e8620ae5404f1b8603a0d36f18cd38 received data dumped as job5.6.0.9728.received expected data dumped as job5.6.0.9728.expected job0: (g=0): rw=randrw, bs=512-64K/512-64K, ioengine=sync, iodepth=1 job1: (g=0): rw=randrw, bs=512-64K/512-64K, ioengine=sync, iodepth=1 job2: (g=0): rw=randrw, bs=512-64K/512-64K, ioengine=sync, iodepth=1 job3: (g=0): rw=randrw, bs=512-64K/512-64K, ioengine=sync, iodepth=1 job4: (g=0): rw=randrw, bs=512-64K/512-64K, ioengine=sync, iodepth=1 job5: (g=0): rw=randrw, bs=512-64K/512-64K, ioengine=sync, iodepth=1 job6: (g=0): rw=randrw, bs=512-64K/512-64K, ioengine=sync, iodepth=1 job7: (g=0): rw=randrw, bs=512-64K/512-64K, ioengine=sync, iodepth=1 job8: (g=0): rw=randrw, bs=512-64K/512-64K, ioengine=sync, iodepth=1 job9: (g=0): rw=randrw, bs=512-64K/512-64K, ioengine=sync, iodepth=1 fio 1.59 Starting 10 processes job0: Laying out IO file(s) (1000 file(s) / 4997MB) job1: Laying out IO file(s) (1000 file(s) / 4921MB) job2: Laying out IO file(s) (1000 file(s) / 5140MB) job3: Laying out IO file(s) (1000 file(s) / 5086MB) job4: Laying out IO file(s) (1000 file(s) / 4869MB) job5: Laying out IO file(s) (1000 file(s) / 5106MB) job6: Laying out IO file(s) (1000 file(s) / 4980MB) job7: Laying out IO file(s) (1000 file(s) / 5052MB) job8: Laying out IO file(s) (1000 file(s) / 5075MB) job9: Laying out IO file(s) (1000 file(s) / 4964MB) fio: pid=2684, got signal=6 job0: (groupid=0, jobs=1): err=84 (file:io_u.c:1425, func=io_u_sync_complete, error=Invalid or incomplete multibyte or wide character): pid=2679 read : io=2425.2MB, bw=139606 B/s, iops=9 , runt=18215098msec clat (usec): min=1 , max=1516.9K, avg=28821.71, stdev=53801.70 lat (usec): min=1 , max=1516.9K, avg=28823.38, stdev=53801.36 bw (KB/s) : min=0, max= 1677, per=15058.66%, avg=150.59, stdev=127.61 write: io=2422.1MB, bw=139478 B/s, iops=9 , runt=18215071msec clat (usec): min=12 , max=11509K, avg=74324.21, stdev=123810.94 lat (usec): min=12 , max=11509K, avg=74324.85, stdev=123810.96 bw (KB/s) : min=0, max= 1541, per=14368.19%, avg=143.68, stdev=89.16 cpu : usr=0.08%, sys=0.21%, ctx=501180, majf=0, minf=4422 IO depths: 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit
Re: fio reports data corruption with btrfs
On Wed, Jun 27, 2012 at 02:15:59AM -0600, Alex Lyakas wrote: > Hi Josef, > I have rerun the test with btrfs-next master branch. fio reported > mismatched blocks again. Mount options were the same (-o > noatime,nodatacow). > Ok I'll try running it again here locally, I didn't realize it was nodatacow. In the meantime can you try a newer version of fio, I'm running fio-2.0.6-1.fc18.x86_64 but I think git is even newer than that. Thanks, Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: fio reports data corruption with btrfs
Hi Josef, which branch from your (or another?) repo should I try this on: remotes/origin/HEAD -> origin/master remotes/origin/enospc3c78455 Btrfs: fall back to non-inline if we don't have enough space remotes/origin/for-chris 7e9ce65 Btrfs: hold a ref on the inode during writepages remotes/origin/master1573317 Btrfs: use helper function to simplify code Thanks, Alex. On Wed, Jun 27, 2012 at 4:46 PM, Josef Bacik wrote: > On Wed, Jun 27, 2012 at 02:15:59AM -0600, Alex Lyakas wrote: >> Hi Josef, >> I have rerun the test with btrfs-next master branch. fio reported >> mismatched blocks again. Mount options were the same (-o >> noatime,nodatacow). >> > > Ok I'll try running it again here locally, I didn't realize it was nodatacow. > In the meantime can you try a newer version of fio, I'm running > fio-2.0.6-1.fc18.x86_64 but I think git is even newer than that. Thanks, > > Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: fio reports data corruption with btrfs
On Mon, Jul 02, 2012 at 05:48:43AM -0600, Alex Lyakas wrote: > Hi Josef, > which branch from your (or another?) repo should I try this on: > Master, and make sure you've checked out a more recent fio from git to make sure it's not a bug in fio. Thanks, Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: fio reports data corruption with btrfs
Hi Josef, tested with your "master" branch, commit 157331741ba010ffcb2212b88342fb21ae140636. fio compiled from tag fio-2.0.8 (commit cf9a74c8bd63d9db5256f1362885c740e11a1fe5). Result: some kernel warnings about hung tasks, fio segfault and mismatch errors. Below are some outputs... I will try some more tests, changing different components. Alex. Kernel log: Jul 3 00:02:12 vsa-0018-vc-1 kernel: [ 9000.740135] INFO: task df:3113 blocked for more than 120 seconds. Jul 3 00:02:12 vsa-0018-vc-1 kernel: [ 9000.741878] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jul 3 00:02:12 vsa-0018-vc-1 kernel: [ 9000.742881] df D 8180cae0 0 3113 3112 0x Jul 3 00:02:12 vsa-0018-vc-1 kernel: [ 9000.742887] 88004668fcb8 0086 880119b72e28 88011fc13930 Jul 3 00:02:12 vsa-0018-vc-1 kernel: [ 9000.742891] 88004668ffd8 88004668ffd8 88004668ffd8 000138c0 Jul 3 00:02:12 vsa-0018-vc-1 kernel: [ 9000.742894] 81c13440 8801194544d0 88011fc138c0 7fff Jul 3 00:02:12 vsa-0018-vc-1 kernel: [ 9000.742896] Call Trace: Jul 3 00:02:12 vsa-0018-vc-1 kernel: [ 9000.742912] [] schedule+0x29/0x70 Jul 3 00:02:12 vsa-0018-vc-1 kernel: [ 9000.742916] [] schedule_timeout+0x2a5/0x320 Jul 3 00:02:12 vsa-0018-vc-1 kernel: [ 9000.742920] [] wait_for_common+0xdf/0x180 Jul 3 00:02:12 vsa-0018-vc-1 kernel: [ 9000.742940] [] ? __sync_filesystem+0x90/0x90 Jul 3 00:02:12 vsa-0018-vc-1 kernel: [ 9000.742952] [] ? try_to_wake_up+0x200/0x200 Jul 3 00:02:12 vsa-0018-vc-1 kernel: [ 9000.742956] [] ? __sync_filesystem+0x90/0x90 Jul 3 00:02:12 vsa-0018-vc-1 kernel: [ 9000.742960] [] wait_for_completion+0x1d/0x20 Jul 3 00:02:12 vsa-0018-vc-1 kernel: [ 9000.742963] [] writeback_inodes_sb_nr+0x7d/0xa0 Jul 3 00:02:12 vsa-0018-vc-1 kernel: [ 9000.742967] [] writeback_inodes_sb+0x2e/0x40 Jul 3 00:02:12 vsa-0018-vc-1 kernel: [ 9000.742970] [] __sync_filesystem+0x4e/0x90 Jul 3 00:02:12 vsa-0018-vc-1 kernel: [ 9000.742973] [] sync_one_sb+0x1f/0x30 Jul 3 00:02:12 vsa-0018-vc-1 kernel: [ 9000.742984] [] iterate_supers+0x101/0x110 Jul 3 00:02:12 vsa-0018-vc-1 kernel: [ 9000.742988] [] sys_sync+0x30/0x70 Jul 3 00:02:12 vsa-0018-vc-1 kernel: [ 9000.742993] [] system_call_fastpath+0x16/0x1b Jul 3 00:04:12 vsa-0018-vc-1 kernel: [ 9120.740290] INFO: task df:3113 blocked for more than 120 seconds. Jul 3 00:04:12 vsa-0018-vc-1 kernel: [ 9120.741252] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jul 3 00:04:12 vsa-0018-vc-1 kernel: [ 9120.742282] df D 8180cae0 0 3113 3112 0x Jul 3 00:04:12 vsa-0018-vc-1 kernel: [ 9120.742299] 88004668fcb8 0086 880119b72e28 88011fc13930 Jul 3 00:04:12 vsa-0018-vc-1 kernel: [ 9120.742303] 88004668ffd8 88004668ffd8 88004668ffd8 000138c0 Jul 3 00:04:12 vsa-0018-vc-1 kernel: [ 9120.742307] 81c13440 8801194544d0 88011fc138c0 7fff Jul 3 00:04:12 vsa-0018-vc-1 kernel: [ 9120.742308] Call Trace: Jul 3 00:04:12 vsa-0018-vc-1 kernel: [ 9120.742324] [] schedule+0x29/0x70 Jul 3 00:04:12 vsa-0018-vc-1 kernel: [ 9120.742328] [] schedule_timeout+0x2a5/0x320 Jul 3 00:04:12 vsa-0018-vc-1 kernel: [ 9120.742332] [] wait_for_common+0xdf/0x180 Jul 3 00:04:12 vsa-0018-vc-1 kernel: [ 9120.742340] [] ? __sync_filesystem+0x90/0x90 Jul 3 00:04:12 vsa-0018-vc-1 kernel: [ 9120.742347] [] ? try_to_wake_up+0x200/0x200 Jul 3 00:04:12 vsa-0018-vc-1 kernel: [ 9120.742350] [] ? __sync_filesystem+0x90/0x90 Jul 3 00:04:12 vsa-0018-vc-1 kernel: [ 9120.742354] [] wait_for_completion+0x1d/0x20 Jul 3 00:04:12 vsa-0018-vc-1 kernel: [ 9120.742358] [] writeback_inodes_sb_nr+0x7d/0xa0 Jul 3 00:04:12 vsa-0018-vc-1 kernel: [ 9120.742361] [] writeback_inodes_sb+0x2e/0x40 Jul 3 00:04:12 vsa-0018-vc-1 kernel: [ 9120.742364] [] __sync_filesystem+0x4e/0x90 Jul 3 00:04:12 vsa-0018-vc-1 kernel: [ 9120.742368] [] sync_one_sb+0x1f/0x30 Jul 3 00:04:12 vsa-0018-vc-1 kernel: [ 9120.742373] [] iterate_supers+0x101/0x110 Jul 3 00:04:12 vsa-0018-vc-1 kernel: [ 9120.742376] [] sys_sync+0x30/0x70 Jul 3 00:04:12 vsa-0018-vc-1 kernel: [ 9120.742381] [] system_call_fastpath+0x16/0x1b Jul 3 00:06:12 vsa-0018-vc-1 kernel: [ 9240.740133] INFO: task df:3113 blocked for more than 120 seconds. Jul 3 00:06:12 vsa-0018-vc-1 kernel: [ 9240.741191] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jul 3 00:06:12 vsa-0018-vc-1 kernel: [ 9240.742103] df D 8180cae0 0 3113 3112 0x Jul 3 00:06:12 vsa-0018-vc-1 kernel: [ 9240.742109] 88004668fcb8 0086 880119b72e28 88011fc13930 Jul 3 00:06:12 vsa-0018-vc-1 kernel: [ 9240.742113] 88004668ffd8 88004668ffd8 88004668ffd8 000
