On Thu, Jan 12, 2017 at 12:02 PM, Mark Greenall <m.green...@iontrading.com> wrote: > Firstly, thanks @Yaniv and thanks @Nir for your responses. > > @Yaniv, in answer to this: > >>> Why do you have 1 SD per VM? > > It's a combination of performance and ease of management. We ran some IO > tests with various configurations and settled on this one for a balance of > reduced IO contention and ease of management. If there is a better > recommended way of handling these then I'm all ears. If you believe having a > large amount of storage domains adds to the problem then we can also review > the setup. > >>> Can you try and disable (mask) the lvmetad service on the hosts and see if >>> it improves matters? > > Disabled and masked the lvmetad service and tried again this morning. It > seemed to be less of a load / quicker getting the initial activation of the > host working but the end result was still the same. Just under 10 minutes > later the node went non-operational and the cycle began again. By 09:27 we > had the high CPU load and repeating lvm cycle. > > Host Activation: 09:06 > Host Up: 09:08 > Non-Operational: 09:16 > LVM Load: 09:27 > Host Reboot: 09:30 > > From yesterday and today I've attached messages, sanlock.log and > multipath.conf files too. Although I'm not sure the messages file will be of > much use as it looks like log rate limiting kicked in and supressed messages > for the duration of the process. I'm booted off the kernel with debugging but > maybe that's generating too much info? Let me know if you want me to change > anything here to get additional information. > > As added configuration information we also have the following settings from > the Equallogic and Linux install guide: > > /etc/sysctl.conf: > > # Prevent ARP Flux for multiple NICs on the same subnet: > net.ipv4.conf.all.arp_ignore = 1 > net.ipv4.conf.all.arp_announce = 2 > # Loosen RP Filter to alow multiple iSCSI connections > net.ipv4.conf.all.rp_filter = 2 > > > And the following /lib/udev/rules.d/99-eqlsd.rules: > > #----------------------------------------------------------------------------- > # Copyright (c) 2010-2012 by Dell, Inc. > # > # All rights reserved. This software may not be copied, disclosed, > # transferred, or used except in accordance with a license granted > # by Dell, Inc. This software embodies proprietary information > # and trade secrets of Dell, Inc. > # > #----------------------------------------------------------------------------- > # > # Various Settings for Dell Equallogic disks based on Dell Optimizing SAN > Environment for Linux Guide > # > # Modify disk scheduler mode to noop > ACTION=="add|change", SUBSYSTEM=="block", ATTRS{vendor}=="EQLOGIC", > RUN+="/bin/sh -c 'echo noop > /sys/${DEVPATH}/queue/scheduler'" > # Modify disk timeout value to 60 seconds > ACTION!="remove", SUBSYSTEM=="block", ATTRS{vendor}=="EQLOGIC", RUN+="/bin/sh > -c 'echo 60 > /sys/%p/device/timeout'"
This timeout may cause large timeouts in vdsm in commands accessing storage, it may cause timeouts in various flows, and may cause your domain to become inactive - since you set this for all domains, it may cause the entire host to become non-operational. I recommend to remove this rule. > # Modify read ahead value to 1024 > ACTION!="remove", SUBSYSTEM=="block", ATTRS{vendor}=="EQLOGIC", RUN+="/bin/sh > -c 'echo 1024 > /sys/${DEVPATH}/bdi/read_ahead_kb'" In your multipath.conf, I see that you changed lot of the defaults recommended by ovirt: defaults { deferred_remove yes dev_loss_tmo 30 fast_io_fail_tmo 5 flush_on_last_del yes max_fds 4096 no_path_retry fail polling_interval 5 user_friendly_names no } You are using: defaults { You are not using "deferred_remove", so you get the default value ("no"). Do you have any reason to change this? You are not using "dev_loss_tmo", so you get the default value Do you have any reason to change this? You are not using "fast_io_fail_tmo", so you will get the default value (hopefully 5). Do you have any reason to change this? You are not using "flush_on_last_del " - any reason to change this? failback immediate max_fds 8192 no_path_retry fail I guess these are the settings recommended for your storage? path_checker tur path_grouping_policy multibus path_selector "round-robin 0" polling_interval 10 This will means multipathd will check paths every 10-40 seconds. You should use the default 5, which cause multipathd to check every 5-20 seconds. rr_min_io 10 rr_weight priorities user_friendly_names no } Also you are mixing defaults and settings that you need for your specific devices. You should leave the default without change, and create a device section for your device: devices { device { vendor XXX product YYY # ovirt specific settings deferred_remove yes dev_loss_tmo 30 fast_io_fail_tmo 5 flush_on_last_del yes no_path_retry fail polling_interval 5 user_friendly_names no # device specific settings max_fds 8192 path_checker tur path_grouping_policy multibus path_selector "round-robin 0" } } Note that you must copy ovirt defaults into the device section, otherwise you will get multipathd builtin defaults, which are not the same. Can you share also the output of: multipath -ll In this command you can see the name of vendor and product. Using these names, find the effective configuration of your multipath devices using this command: multipathd show config If the the device is not listed in the output, you are using the defaults. Please share here the configuration for your device or the defaults from the output of multiapthd show config. For example, here is my test storage: # multipath -ll 3600140549f3b93968d440ac9129d124f dm-11 LIO-ORG ,target1-12 size=50G features='0' hwhandler='0' wp=rw `-+- policy='service-time 0' prio=1 status=active `- 22:0:0:12 sdi 8:128 active ready running multipathd does not have any setting for LIO-ORG, so we get the defaults: defaults { verbosity 2 polling_interval 5 max_polling_interval 20 reassign_maps "yes" multipath_dir "/lib64/multipath" path_selector "service-time 0" path_grouping_policy "failover" uid_attribute "ID_SERIAL" prio "const" prio_args "" features "0" path_checker "directio" alias_prefix "mpath" failback "manual" rr_min_io 1000 rr_min_io_rq 1 max_fds 4096 rr_weight "uniform" no_path_retry "fail" queue_without_daemon "no" flush_on_last_del "yes" user_friendly_names "no" fast_io_fail_tmo 5 dev_loss_tmo 30 bindings_file "/etc/multipath/bindings" wwids_file /etc/multipath/wwids log_checker_err always find_multipaths no retain_attached_hw_handler no detect_prio no hw_str_match no force_sync no deferred_remove yes ignore_new_boot_devs no skip_kpartx no config_dir "/etc/multipath/conf.d" delay_watch_checks no delay_wait_checks no retrigger_tries 3 retrigger_delay 10 missing_uev_wait_timeout 30 new_bindings_in_boot no } Nir _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users