Public bug reported:
We upgraded our 12.04 LTS on 9 July at around 18:30. Immediately after
the reboot, the I/O on the / partition (sda) was extremely high. This
was causing sluggish responsiveness on the NFS server who's exported
directory is on a different file system (sdb) and LUN (disk).
I investigated the problem and found using iotop that the process "jbd2/sda2-8"
was responsible for an extremely high number of I/O operations
Total DISK READ: 0.00 B/s | Total DISK WRITE: 13.33 M/
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
293 be/3 root 0.00 B/s 0.00 B/s 0.00 % 39.16 % [jbd2/sda2-8]
I turned on file system debugging using
echo 1 > /sys/kernel/debug/tracing/events/ext4/ext4_sync_file_enter/enable
echo 1 >/sys/kernel/debug/tracing/events/jbd2/jbd2_run_stats/enable
and found that one inode was being massively addressed.
jbd2/sda2-8-293 [000] 260587.952474: jbd2_run_stats: dev 8,2 tid
42050430 wait 0 running 0 locked 0 flushing 0 logging 0 handle_count 2 blocks 7
blocks_logged 8
nfsd-1332 [000] 260587.953142: ext4_sync_file_enter: dev 8,2 ino
150 parent 16987 datasync 0
This inode (150) belongs to /var/lib/nfs/v4recovery
Further investigation showed in dmesg that shortly after booting the directory
/var/lib/nfs/v4recovery couldn't be written to:
[ 99.020861] NFSD: failed to write recovery record (err -17); please check
that /var/lib/nfs/v4recovery exists and is writeable
[ 99.089156] NFSD: failed to write recovery record (err -17); please check
that /var/lib/nfs/v4recovery exists and is writeable
[ 99.189010] NFSD: failed to write recovery record (err -17); please check
that /var/lib/nfs/v4recovery exists and is writeable
I have tried deleting and recreating the directory
/var/lib/nfs/v4recovery with 777 permissions however this did not solve
the problem. These messages were still produced even after a reboot.
The I/O operations per second are sometimes in excess of 1000 and
typically around 500-750. This is completely different behaviour to the
previous kernel where the I/O operations were in the 5-20 I/O operations
per second with peaks around 30. I will attach a graph from our central
EMC storage system of the LUN for a graphical view of before and after
the update.
There were no changes in the parameters or shares to the NFS server and
I have not been able to find any documentation about parameters that we
should change to address such a problem so I can only conclude with this
behaviour that this is a bug of some decsription.
Additional information about our system:
$ uname -a
Linux wsps428 3.2.0-87-generic #125-Ubuntu SMP Fri Jun 19 08:25:10 UTC 2015
x86_64 x86_64 x86_64 GNU/Linux
$ cat /proc/version_signature
Ubuntu 3.2.0-87.125-generic 3.2.69
$ lsb_release -rd
Description: Ubuntu 12.04.5 LTS
Release: 12.04
$ apt-cache policy nfs-server
nfs-server:
Installed: (none)
Candidate: (none)
Version table:
I will attach a dmesg output.
** Affects: ubuntu
Importance: Undecided
Status: New
** Attachment added: "dmesg.out"
https://bugs.launchpad.net/bugs/1473948/+attachment/4428262/+files/dmesg.out
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1473948
Title:
After most recent upgrade to 3.2.0-87-generic, nfs server process has
extremely high I/O to /var/lib/nfs/v4recovery
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+bug/1473948/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
