Bug#827322: multipath-tools: Not all paths discovered when large number of paths are present
On 06/25/2016 02:16 AM, Ritesh Raj Sarraf wrote: > On Thu, 2016-06-23 at 16:50 -0600, Andrew Patterson wrote: >> +# Save boot > setting for aio-max-nr. >> +BOOT_AIO_MAX_NR=$(cat /proc/sys/fs/aio-max-nr) >> + >> if [ -f /etc/multipath-tools-boot.conf ]; then >> . /etc/multipath-tools-boot.conf >> else >> MULTIPATH_TOOLS_MODULES="dm-mod dm-multipath" >> + MULTIPATH_TOOLS_MIN_AIO_MAX_NR=$BOOT_AIO_MAX_NR >> fi >> > > I'd let the user define the AIO_MAX value explicitly in /etc/default/multipath- > tools. And use the sysctl/procfs option as a fallback. > > What do you say ? > > That was where I went initially, but I thought that means that the user now needs to configure aio-max-nr in two places, once in sysctl.conf and once in /etc/default/multipath-tools (for when running multipath tools by hand or in scripts). I thought this solution resulted in less overall system configuration. The /etc/default/multipath-tools is more explicit, so it does have advantages too. Andrew -- Andrew Patterson Hewlett-Packard Enterprise
Bug#827322: multipath-tools: Not all paths discovered when large number of paths are present
On Thu, 2016-06-23 at 16:50 -0600, Andrew Patterson wrote: > +# Save boot setting for aio-max-nr. > +BOOT_AIO_MAX_NR=$(cat /proc/sys/fs/aio-max-nr) > + > if [ -f /etc/multipath-tools-boot.conf ]; then > . /etc/multipath-tools-boot.conf > else > MULTIPATH_TOOLS_MODULES="dm-mod dm-multipath" > +MULTIPATH_TOOLS_MIN_AIO_MAX_NR=$BOOT_AIO_MAX_NR > fi > I'd let the user define the AIO_MAX value explicitly in /etc/default/multipath- tools. And use the sysctl/procfs option as a fallback. What do you say ? -- Ritesh Raj Sarraf | http://people.debian.org/~rrs Debian - The Universal Operating System signature.asc Description: This is a digitally signed message part
Bug#827322: multipath-tools: Not all paths discovered when large number of paths are present
On Mon, 20 Jun 2016 16:40:29 -0600 Andrew Patterson
wrote:
> On Thu, 16 Jun 2016 15:19:22 +0530 Ritesh Raj Sarraf wrote:
> > Control: tag -1 +pending
> >
> > On Wed, 2016-06-15 at 13:03 -0600, Andrew Patterson wrote:
> > > For most users this is not an issue. How many systems are going to use
> > > that many LUNs?
> > >
> >
> > I agree.
> >
> > > One counter-example is using one or more FC LUNs per VM guest on a
> > > host. In such a case, one can just modify /etc/sysctl.conf to increase
> > > the setting.
> >
> > Yes. Most hypervisors may want this. But again, like you said, most users
> > do not
> > map that many LUNs. And recommended practice is to have a VM backed by a
> > file
> > backend mostly.
>
>
> OpenStack/cinder can (and is) be uses to provision VMs with networked
> drives. It is pretty common for cloud applications. I don't know what
> a practical limit for the number of volume per VMs per host should be.
>
>
> >
> > > We can run into problems when using
> > > multipath-tools-boot. In this case, we use the default kernel settings
> > > for aio-max-nr since multipath is run from the initramfs. I have
> > > worked around this by modifying the script in
> > > /usr/share/initramfs/scripts/local-top/multipath to temporarilly set
> > > aio-max-nr to 1048576 while doing discovery.
> >
> >
> > So, I'll just add this to README.Debian. There's not much that we can do
> > beyond
> > documenting such behavior.
>
> I will work on patch to have a user-configurable setting of aio-max-nr
> in the initramfs. You can then decide if it is worth it to apply it.
>
> In the meantime, the README.Debian change sounds fine to me.
>
Here is a patch that copies the fs.aio-max-nr setting to the
initramfs. There is more to do to solve the problem with booting with
many LUNs, but this is a prerequisite for completely solving that
issue. This patch assumes that the patch in:
https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=827407;filename=multipath-tools-827407.patch;msg=30
has been applied. I can rework it if that one is not accepted.
Andrew
--
Andrew Patterson
Hewlett-Packard Enterprise
commit e68651ca8b6a9c2b8afee787b3e5221b402846fd
Author: Andrew Patterson
Date: Thu Jun 23 22:34:16 2016 +
Use fs.aio-max-nr setting in initramfs
We sometimes need to increase the fs.aio-max-nr from the kernel default
value when running multipath discovery in the initramfs with large numbers
of LUNs. This patch copies the setting there and temporarily changes it
while performing discovery.
diff --git a/debian/initramfs/hooks b/debian/initramfs/hooks
index 94cb7e8..a29dbac 100644
--- a/debian/initramfs/hooks
+++ b/debian/initramfs/hooks
@@ -63,4 +63,4 @@ for x in $MULTIPATH_TOOLS_MODULES; do
done
echo "MULTIPATH_TOOLS_HW_HANDLERS=\"$MULTIPATH_TOOLS_HW_HANDLERS\"" > $DESTDIR/etc/multipath-tools-boot.conf
echo "MULTIPATH_TOOLS_MODULES=\"$MULTIPATH_TOOLS_MODULES\"" >> $DESTDIR/etc/multipath-tools-boot.conf
-
+echo "MULTIPATH_TOOLS_MIN_AIO_MAX_NR=$(sysctl fs.aio-max-nr | cut -f3 -d ' ')" >> $DESTDIR/etc/multipath-tools-boot.conf
diff --git a/debian/initramfs/local-top b/debian/initramfs/local-top
index 99e4222..67698bd 100644
--- a/debian/initramfs/local-top
+++ b/debian/initramfs/local-top
@@ -31,10 +31,14 @@ if [ ! -e /sbin/multipath ]; then
exit 0
fi
+# Save boot setting for aio-max-nr.
+BOOT_AIO_MAX_NR=$(cat /proc/sys/fs/aio-max-nr)
+
if [ -f /etc/multipath-tools-boot.conf ]; then
. /etc/multipath-tools-boot.conf
else
MULTIPATH_TOOLS_MODULES="dm-mod dm-multipath"
+MULTIPATH_TOOLS_MIN_AIO_MAX_NR=$BOOT_AIO_MAX_NR
fi
verbose && log_begin_msg "Loading multipath modules"
@@ -47,6 +51,12 @@ for module in ${MULTIPATH_TOOLS_MODULES}; do
done
verbose && log_end_msg
+
+# Adjust aio-max-nr if too low.
+if [ $BOOT_AIO_MAX_NR -le $MULTIPATH_TOOLS_MIN_AIO_MAX_NR ]; then
+echo "$MULTIPATH_TOOLS_MIN_AIO_MAX_NR" > /proc/sys/fs/aio-max-nr
+fi
+
verbose && log_begin_msg "Discovering multipaths"
/sbin/multipath -v $VERBOSITY
verbose && log_end_msg
@@ -59,6 +69,9 @@ if [ -x /sbin/udevadm ]; then
/sbin/udevadm settle --timeout=10
fi
+# Restore aio-max-nr.
+echo "$BOOT_AIO_MAX_NR" > /proc/sys/fs/aio-max-nr
+
maybe_break post-multipath
exit 0
diff --git a/debian/multipath-tools.README.Debian b/debian/multipath-tools.README.Debian
index c1f50a2..2665046 100644
--- a/debian/multipath-tools.README.Debian
+++ b/debian/multipath-tools.README.Debian
@@ -35,7 +35,8 @@ To make this setting persistent a line may be added to /etc/sysctl.conf:
Consult appropriate application and operating system tuning recommendations for
guidance on appropriate values for this parameter.
-
+If using multipath-tools-boot, remember to run update-initramfs so the
+adjusted aio-max-nr will also be set in the initramfs.
What are these weird numbers in /dev/mapper?
Bug#827322: multipath-tools: Not all paths discovered when large number of paths are present
On Thu, 16 Jun 2016 15:19:22 +0530 Ritesh Raj Sarraf wrote: > Control: tag -1 +pending > > On Wed, 2016-06-15 at 13:03 -0600, Andrew Patterson wrote: > > For most users this is not an issue. How many systems are going to use > > that many LUNs? > > > > I agree. > > > One counter-example is using one or more FC LUNs per VM guest on a > > host. In such a case, one can just modify /etc/sysctl.conf to increase > > the setting. > > Yes. Most hypervisors may want this. But again, like you said, most users do > not > map that many LUNs. And recommended practice is to have a VM backed by a file > backend mostly. OpenStack/cinder can (and is) be uses to provision VMs with networked drives. It is pretty common for cloud applications. I don't know what a practical limit for the number of volume per VMs per host should be. > > > We can run into problems when using > > multipath-tools-boot. In this case, we use the default kernel settings > > for aio-max-nr since multipath is run from the initramfs. I have > > worked around this by modifying the script in > > /usr/share/initramfs/scripts/local-top/multipath to temporarilly set > > aio-max-nr to 1048576 while doing discovery. > > > So, I'll just add this to README.Debian. There's not much that we can do > beyond > documenting such behavior. I will work on patch to have a user-configurable setting of aio-max-nr in the initramfs. You can then decide if it is worth it to apply it. In the meantime, the README.Debian change sounds fine to me. -- Andrew Patterson Hewlett-Packard Enterprise
Bug#827322: multipath-tools: Not all paths discovered when large number of paths are present
Control: tag -1 +pending On Wed, 2016-06-15 at 13:03 -0600, Andrew Patterson wrote: > For most users this is not an issue. How many systems are going to use > that many LUNs? > I agree. > One counter-example is using one or more FC LUNs per VM guest on a > host. In such a case, one can just modify /etc/sysctl.conf to increase > the setting. Yes. Most hypervisors may want this. But again, like you said, most users do not map that many LUNs. And recommended practice is to have a VM backed by a file backend mostly. > We can run into problems when using > multipath-tools-boot. In this case, we use the default kernel settings > for aio-max-nr since multipath is run from the initramfs. I have > worked around this by modifying the script in > /usr/share/initramfs/scripts/local-top/multipath to temporarilly set > aio-max-nr to 1048576 while doing discovery. So, I'll just add this to README.Debian. There's not much that we can do beyond documenting such behavior. -- Ritesh Raj Sarraf | http://people.debian.org/~rrs Debian - The Universal Operating System signature.asc Description: This is a digitally signed message part
Bug#827322: multipath-tools: Not all paths discovered when large number of paths are present
On Thu, 16 Jun 2016 00:15:37 +0530 Ritesh Raj Sarraf wrote: > On Wed, 2016-06-15 at 11:12 -0600, Andrew Patterson wrote: > > > > Thanks for reminding me about max_fds. At boot: > > > > # cat /proc/sys/fs/nr_open > > 1048576 > > > > and > > > > # multipath -t | grep max_fds > > Â Â Â Â Â Â Â max_fds "max" > > > > So this should already be set to 1048576. I tried the test again after > > setting max_fd 1048576 in /etc/multipath.conf and got the same > > behavior. > > This dates way back. 2009 is a memorable year. > > > https://www.redhat.com/archives/dm-devel/2009-July/msg00020.html > > Quoting: > > + > +4. I see a lot of "io_setup failed" message using the directio checker > +== > + > +The directio path checker makes use of the asynchronous I/O API (aio) > provided > +by modern Linux systems. Asynchronous I/O allows an application to submit I/O > +requests asynchronously and be notified later of their completion status. To > +allow this, we must allocate an asynchronous I/O context (an object of type > +aio_context_t) and this task is handled by the io_setup system call. > + > +A system wide limit on the number of AIO contexts that may be active > +simultaneously is imposed via the aio-max-nr sysctl parameter. > + > +Once this limit has been reached further calls to io_setup will fail with the > +error number EAGAIN leading to the "io_setup failed" messages seen for e.g. > when > +running "multipath -ll". > + > +To avoid this problem the number of available aio contexts should be > increased > +by setting the aio-max-nr parameter. This can be set on a one-time basis via > the > +/proc file system, for e.g.: > + > +Â Â Â Â Â Â Â Â # echo 131072 > /proc/sys/fs/aio-max-nr > + > +Doubles the number of available contexts from the default value of 65536. > + > +To make this setting persistent a line may be added to /etc/sysctl.conf: > + > +Â Â Â Â Â Â Â Â fs.aio-max-nr = 131072 > + > +Consult appropriate application and operating system tuning recommendations > for > +guidance on appropriate values for this parameter. > > > -- > Ritesh Raj Sarraf | http://people.debian.org/~rrs For most users this is not an issue. How many systems are going to use that many LUNs? One counter-example is using one or more FC LUNs per VM guest on a host. In such a case, one can just modify /etc/sysctl.conf to increase the setting. We can run into problems when using multipath-tools-boot. In this case, we use the default kernel settings for aio-max-nr since multipath is run from the initramfs. I have worked around this by modifying the script in /usr/share/initramfs/scripts/local-top/multipath to temporarilly set aio-max-nr to 1048576 while doing discovery. -- Andrew Patterson Hewlett-Packard Enterprise
Bug#827322: multipath-tools: Not all paths discovered when large number of paths are present
On Wed, 2016-06-15 at 11:12 -0600, Andrew Patterson wrote: > > Thanks for reminding me about max_fds. At boot: > > # cat /proc/sys/fs/nr_open > 1048576 > > and > > # multipath -t | grep max_fds > max_fds "max" > > So this should already be set to 1048576. I tried the test again after > setting max_fd 1048576 in /etc/multipath.conf and got the same > behavior. This dates way back. 2009 is a memorable year. https://www.redhat.com/archives/dm-devel/2009-July/msg00020.html Quoting: + +4. I see a lot of "io_setup failed" message using the directio checker +== + +The directio path checker makes use of the asynchronous I/O API (aio) provided +by modern Linux systems. Asynchronous I/O allows an application to submit I/O +requests asynchronously and be notified later of their completion status. To +allow this, we must allocate an asynchronous I/O context (an object of type +aio_context_t) and this task is handled by the io_setup system call. + +A system wide limit on the number of AIO contexts that may be active +simultaneously is imposed via the aio-max-nr sysctl parameter. + +Once this limit has been reached further calls to io_setup will fail with the +error number EAGAIN leading to the "io_setup failed" messages seen for e.g. when +running "multipath -ll". + +To avoid this problem the number of available aio contexts should be increased +by setting the aio-max-nr parameter. This can be set on a one-time basis via the +/proc file system, for e.g.: + +# echo 131072 > /proc/sys/fs/aio-max-nr + +Doubles the number of available contexts from the default value of 65536. + +To make this setting persistent a line may be added to /etc/sysctl.conf: + +fs.aio-max-nr = 131072 + +Consult appropriate application and operating system tuning recommendations for +guidance on appropriate values for this parameter. -- Ritesh Raj Sarraf | http://people.debian.org/~rrs Debian - The Universal Operating System signature.asc Description: This is a digitally signed message part
Bug#827322: multipath-tools: Not all paths discovered when large number of paths are present
On 06/15/2016 08:43 AM, Ritesh Raj Sarraf wrote: > Control: tag -1 moreinfo > > On Tue, 2016-06-14 at 17:30 -0600, Andrew Patterson wrote: >> There is no output in dmesg. >> >> Changing /proc/sys/fs/aio-max-nr from the default 65536 to 1048576 >> seems to help. With this change, I can go up to 256 LUNs (1024 paths) >> with no problems. >> >> I have seen similar issues in earlier versions of multipath-tools. > > Thank you for bringing this up, and the pointer. In multipath.conf there was a > directive added for something similar. > > >max_fds Specify the maximum number of file descriptors that > can be opened by multipath and multipathd. This is > equivalent to ulimit -n. A value of max will set this > to the system limit from /proc/sys/fs/nr_open. If this > is not set, the maximum number of open fds is taken > from the calling process. It is usually 1024. To be > safe, this should be set to the maximum number of > paths plus 32, if that number is greated than 1024. > > > It seems to be inline with what aio-max-nr documentation says. > > == > > aio-nr & aio-max-nr: > > aio-nr is the running total of the number of events specified on the > io_setup system call for all currently active aio contexts. If aio-nr > reaches aio-max-nr then io_setup will fail with EAGAIN. Note that > raising aio-max-nr does not result in the pre-allocation or re-sizing > of any kernel data structures. > > == > > > Have you tried it with the "max_fds" option set ? > > Thanks for reminding me about max_fds. At boot: # cat /proc/sys/fs/nr_open 1048576 and # multipath -t | grep max_fds max_fds "max" So this should already be set to 1048576. I tried the test again after setting max_fd 1048576 in /etc/multipath.conf and got the same behavior. -- Andrew Patterson Hewlett-Packard Enterprise
Bug#827322: multipath-tools: Not all paths discovered when large number of paths are present
Control: tag -1 moreinfo On Tue, 2016-06-14 at 17:30 -0600, Andrew Patterson wrote: > There is no output in dmesg. > > Changing /proc/sys/fs/aio-max-nr from the default 65536 to 1048576 > seems to help. With this change, I can go up to 256 LUNs (1024 paths) > with no problems. > > I have seen similar issues in earlier versions of multipath-tools. Thank you for bringing this up, and the pointer. In multipath.conf there was a directive added for something similar. max_fds Specify the maximum number of file descriptors that can be opened by multipath and multipathd. This is equivalent to ulimit -n. A value of max will set this to the system limit from /proc/sys/fs/nr_open. If this is not set, the maximum number of open fds is taken from the calling process. It is usually 1024. To be safe, this should be set to the maximum number of paths plus 32, if that number is greated than 1024. It seems to be inline with what aio-max-nr documentation says. == aio-nr & aio-max-nr: aio-nr is the running total of the number of events specified on the io_setup system call for all currently active aio contexts. If aio-nr reaches aio-max-nr then io_setup will fail with EAGAIN. Note that raising aio-max-nr does not result in the pre-allocation or re-sizing of any kernel data structures. == Have you tried it with the "max_fds" option set ? -- Ritesh Raj Sarraf | http://people.debian.org/~rrs Debian - The Universal Operating System signature.asc Description: This is a digitally signed message part
Bug#827322: multipath-tools: Not all paths discovered when large number of paths are present
Package: multipath-tools
Version: 0.6.1-3
Multipath discovery fails to find some paths when a large number of
paths are used. I used the following to test this:
1. Present 64 FC LUNs to the OS with 4 paths each (256 paths total).
2. Run multipath -F to make sure we don't have any intermediate results.
3. Run the multipath command, e.g., multipath -v2. In the error case, you
will see "io_setup failed" for some devices.
4. Run multipath -l, checking each mulipath device to make sure
all paths are present (a random number will be missing in the error case).
You can run "multipath -l | grep -E '\|-|`- ' | grep active | wc -l)" to
count the active paths.
Example:
Testing on an amd64 system running sid with
multipath-tools/kpartx_0.6.1-3 from experimental. We hav 64 FC LUNs
with 4 paths each (total 256 paths used mutlipath). The kernel version
is 4.6.0-1-amd64.
With the following /etc/multipath.conf file:
defaults {
user_friendly_names "yes"
}
blacklist {
device {
vendor "HP"
product "LOGICAL VOLUME"
}
Here are the devices used (the boot device is blacklisted):
# lsscsi
[0:0:0:0]storage HP P244br 2.52 -
[0:1:0:0]diskHP LOGICAL VOLUME 2.52 /dev/sda
[1:0:0:0]disk3PARdata VV 3212 /dev/sdj
[1:0:0:1]disk3PARdata VV 3212 /dev/sdl
[1:0:0:2]disk3PARdata VV 3212 /dev/sdn
[1:0:0:3]disk3PARdata VV 3212 /dev/sdq
[1:0:0:4]disk3PARdata VV 3212 /dev/sdr
[1:0:0:5]disk3PARdata VV 3212 /dev/sdt
[1:0:0:6]disk3PARdata VV 3212 /dev/sdv
[1:0:0:7]disk3PARdata VV 3212 /dev/sdx
[1:0:0:8]disk3PARdata VV 3212 /dev/sdah
[1:0:0:9]disk3PARdata VV 3212 /dev/sdaj
[1:0:0:10] disk3PARdata VV 3212 /dev/sdal
[1:0:0:11] disk3PARdata VV 3212 /dev/sdan
[1:0:0:12] disk3PARdata VV 3212 /dev/sdap
[1:0:0:13] disk3PARdata VV 3212 /dev/sdar
[1:0:0:14] disk3PARdata VV 3212 /dev/sdat
[1:0:0:15] disk3PARdata VV 3212 /dev/sdav
[1:0:0:16] disk3PARdata VV 3212 /dev/sday
[1:0:0:17] disk3PARdata VV 3212 /dev/sdbs
[1:0:0:18] disk3PARdata VV 3212 /dev/sdbu
[1:0:0:19] disk3PARdata VV 3212 /dev/sdbw
[1:0:0:20] disk3PARdata VV 3212 /dev/sdby
[1:0:0:21] disk3PARdata VV 3212 /dev/sdca
[1:0:0:22] disk3PARdata VV 3212 /dev/sdcc
[1:0:0:23] disk3PARdata VV 3212 /dev/sdce
[1:0:0:24] disk3PARdata VV 3212 /dev/sdcg
[1:0:0:25] disk3PARdata VV 3212 /dev/sdci
[1:0:0:26] disk3PARdata VV 3212 /dev/sdck
[1:0:0:27] disk3PARdata VV 3212 /dev/sdcm
[1:0:0:28] disk3PARdata VV 3212 /dev/sdco
[1:0:0:29] disk3PARdata VV 3212 /dev/sdcq
[1:0:0:30] disk3PARdata VV 3212 /dev/sdcs
[1:0:0:31] disk3PARdata VV 3212 /dev/sdcu
[1:0:0:32] disk3PARdata VV 3212 /dev/sdea
[1:0:0:33] disk3PARdata VV 3212 /dev/sdeb
[1:0:0:34] disk3PARdata VV 3212 /dev/sded
[1:0:0:35] disk3PARdata VV 3212 /dev/sdef
[1:0:0:36] disk3PARdata VV 3212 /dev/sdeh
[1:0:0:37] disk3PARdata VV 3212 /dev/sdej
[1:0:0:38] disk3PARdata VV 3212 /dev/sdel
[1:0:0:39] disk3PARdata VV 3212 /dev/sden
[1:0:0:40] disk3PARdata VV 3212 /dev/sdep
[1:0:0:41] disk3PARdata VV 3212 /dev/sder
[1:0:0:42] disk3PARdata VV 3212 /dev/sdet
[1:0:0:43] disk3PARdata VV 3212 /dev/sdev
[1:0:0:44] disk3PARdata VV 3212 /dev/sdex
[1:0:0:45] disk3PARdata VV 3212 /dev/sdez
[1:0:0:46] disk3PARdata VV 3212 /dev/sdfb
[1:0:0:47] disk3PARdata VV 3212 /dev/sdfc
[1:0:0:48] disk3PARdata VV 3212 /dev/sdff
[1:0:0:49] disk3PARdata VV 3212 /dev/sdfh
[1:0:0:50] disk3PARdata VV 3212 /dev/sdfj
[1:0:0:51] disk3PARdata VV 3212 /dev/sdfl
[1:0:0:52] disk3PARdata VV 3212 /dev/sdfn
[1:0:0:53] disk3PARdata VV 3212 /dev/sdfp
[1:0:0:54] disk3PARdata VV 3212 /dev/sdfr
[1:0:0:55] disk3PARdata VV 3212 /dev/sdft
[1:0:0:56] disk3PARdata VV 3212 /dev/sdfv
[1:0:0:57] disk3PARdata VV 3212 /dev/sdfw
[1:0:0:58] disk3PARdata VV 3212 /dev/sdfz
[1:
