I'm personally not an xfs expert, but let me leave those two comments:
- The test was unfortunately done on an old and outdated kernel from 16.04.1:
4.4.0-36
which got already replaced by several newer kernels
Today we are already on level: 4.4.0.127.133
(and an even newer one is in the 'pipe' - in proposed: 4.4.0.128.134)
$ rmadison --arch=ppc64el linux-generic | grep xenial
linux-generic | 4.4.0.21.22 | xenial | ppc64el
linux-generic | 4.4.0.127.133 | xenial-security | ppc64el
linux-generic | 4.4.0.127.133 | xenial-updates | ppc64el
linux-generic | 4.4.0.128.134 | xenial-proposed | ppc64el
So please always update the kernel to the latest version from
<ubuntu-release>-updates
(or even proposed) before running such kinds of tests.
Just make sure you have access to the updates (Ubuntu archive mirror) and do
(something like):
sudo apt update && sudo apt full-upgrade && sudo apt autoremove
(and in case a kernel update was installed, reboot to activate it)
So please re-run this test on the latest 4.4.0 version (today either
4.4.0.127.133
or even 4.4.0.128.134 from proposed.)
- Does this problem also happen with the (latest) upstream 4.4 kernel?
Knowing this would help to narrow down the issue.
** Changed in: ubuntu-power-systems
Status: New => Triaged
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1775527
Title:
Ubuntu16.04:KVM - xfs_log_force: error -5 returned after stress test
running after a day #345
Status in The Ubuntu-power-systems project:
Triaged
Status in linux package in Ubuntu:
New
Bug description:
== Comment: #0 - Chanh H. Nguyen <chngu...@us.ibm.com> - 2016-09-08 15:01:30
==
We have installed 16.04.1 on our guest and running IO stress test against via
virtual qlogic disk. After running around 8 hrs, this error popping up
"xfs_imap_to_bp: xfs_trans_read_buf() returned error -5" then with all of these
errors "xfs_log_force: error -5 returned" fill up all the log.
Jul 19 19:03:41 tinyg2 kernel: [65047.974604] XFS (vdb9): metadata I/O error:
block 0x2f140 ("xfs_trans_read_buf_map") error 5 numblks 32
Jul 19 19:03:41 tinyg2 kernel: [65047.975885] XFS (vdb9): xfs_imap_to_bp:
xfs_trans_read_buf() returned error -5.
Jul 19 19:03:41 tinyg2 kernel: [65047.976069] XFS (vdb9):
xfs_do_force_shutdown(0x8) called from line 3433 of file
/build/linux-JQcyrX/linux-4.4.0/fs/xfs/xfs_inode.c. Return address =
0xd0000000037951f4
Jul 19 19:03:41 tinyg2 kernel: [65047.976457] XFS (vdb9): metadata I/O error:
block 0x703d54 ("xlog_iodone") error 5 numblks 64
Jul 19 19:03:41 tinyg2 kernel: [65047.976662] XFS (vdb9):
xfs_do_force_shutdown(0x2) called from line 1197 of file
/build/linux-JQcyrX/linux-4.4.0/fs/xfs/xfs_log.c. Return address =
0xd0000000037a4c00
Jul 19 19:03:41 tinyg2 kernel: [65047.976995] XFS (vdb9): Log I/O Error
Detected. Shutting down filesystem
Jul 19 19:03:41 tinyg2 kernel: [65047.977114] XFS (vdb9): Please umount the
filesystem and rectify the problem(s)
follow with all of these error that quickly fill up dmesg log....
Jul 19 19:04:37 tinyg2 kernel: [65103.515386] XFS (vdb9): xfs_log_force:
error -5 returned.
Jul 19 19:05:07 tinyg2 kernel: [65133.595369] XFS (vdb9): xfs_log_force:
error -5 returned.
Jul 19 19:05:37 tinyg2 kernel: [65163.679358] XFS (vdb9): xfs_log_force:
error -5 returned.
Jul 19 19:06:07 tinyg2 kernel: [65193.755383] XFS (vdb9): xfs_log_force:
error -5 returned.
Jul 19 19:06:37 tinyg2 kernel: [65223.835396] XFS (vdb9): xfs_log_force:
error -5 returned.
Jul 19 19:06:51 tinyg2 kernel: [65237.091455] EXT4-fs (loop1): mounting ext2
file system using the ext4 subsystem
Jul 19 19:06:51 tinyg2 kernel: [65237.093219] EXT4-fs (loop1): mounted
filesystem without journal. Opts: (null)
Jul 19 19:06:51 tinyg2 kernel: [65237.623893] Process accounting resumed
Jul 19 19:07:07 tinyg2 kernel: [65253.919361] XFS (vdb9): xfs_log_force:
error -5 returned.
Jul 19 19:07:37 tinyg2 kernel: [65283.995393] XFS (vdb9): xfs_log_force:
error -5 returned.
Jul 19 19:08:08 tinyg2 kernel: [65314.075378] XFS (vdb9): xfs_log_force:
error -5 returned.
Jul 19 19:08:38 tinyg2 kernel: [65344.155397] XFS (vdb9): xfs_log_force:
error -5 returned.
Jul 19 19:09:08 tinyg2 kernel: [65374.235367] XFS (vdb9): xfs_log_force:
error -5 returned.
Jul 19 19:09:38 tinyg2 kernel: [65404.319354] XFS (vdb9): xfs_log_force:
error -5 returned.
Jul 19 19:10:08 tinyg2 kernel: [65434.395368] XFS (vdb9): xfs_log_force:
error -5 returned.
== Comment: #1 - Chanh H. Nguyen <chngu...@us.ibm.com> - 2016-09-08 15:02:27
==
We recently re-created this bug on our new system "cressg2" which is using a
different SAN on kernel -36.
My Ubuntu 16.04.1 guest, cressg2, are running the IO and other stress test,
and it is hitting "xfs_log_force: error -5 returned" problem after it runs the
tests for 15 hours. It is using our lab storage (SVC SAN storage configuation
with access to DS8000 Volume) and switch, compare to tinyg3, so it is using
different storage and fabric switch.
root@cressg2:~# uname -a
Linux cressg2 4.4.0-36-generic #55-Ubuntu SMP Thu Aug 11 18:00:57 UTC 2016
ppc64le ppc64le ppc64le GNU/Linux
[Wed Sep 7 10:26:48 2016] XFS (vdb8): xfs_log_force: error -5 returned.
[Wed Sep 7 10:27:18 2016] XFS (vdb8): xfs_log_force: error -5 returned.
[Wed Sep 7 10:27:32 2016] XFS (vdb8): xfs_log_force: error -5 returned.
[Wed Sep 7 10:27:33 2016] XFS (vdb8): xfs_log_force: error -5 returned.
[Wed Sep 7 10:27:33 2016] XFS (vdb8): xfs_log_force: error -5 returned.
[Wed Sep 7 10:27:48 2016] XFS (vdb8): xfs_log_force: error -5 returned.
[Wed Sep 7 10:27:48 2016] XFS (vdb8): xfs_log_force: error -5 returned.
[Wed Sep 7 10:27:49 2016] XFS (vdb8): xfs_log_force: error -5 returned.
[Wed Sep 7 10:27:56 2016] XFS (vdb8): xfs_log_force: error -5 returned.
[Wed Sep 7 10:28:18 2016] XFS (vdb8): xfs_log_force: error -5 returned.
[Wed Sep 7 10:28:48 2016] XFS (vdb8): xfs_log_force: error -5 returned.
[Wed Sep 7 10:29:18 2016] XFS (vdb8): xfs_log_force: error -5 returned.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu-power-systems/+bug/1775527/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
