[openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master
I am just getting started working with masakari. I am working on master. I have setup Masakari in Devstack (see details at end of email) ... which starts up masakari-engine and masakari-api processes. I have git cloned the masakari-monitors and started them up (roughly) following the instructions at https://github.com/openstack/masakari-monitors . Specifically: # install & startup monitors cd git clone https://github.com/openstack/masakari-monitors.git cd masakari-monitors sudo python setup.py install cd sudo mkdir /etc/masakarimonitors sudo cp ~/masakari-monitors/etc/masakarimonitors/hostmonitor.conf.sample /etc/masakarimonitors/hostmonitor.conf sudo cp ~/masakari-monitors/etc/masakarimonitors/processmonitor.conf.sample /etc/masakarimonitors/processmonitor.conf sudo cp ~/masakari-monitors/etc/masakarimonitors/proc.list.sample /etc/masakarimonitors/proc.list cd ~/masakari-monitors/masakarimonitors/cmd sudo masakari-processmonitor.sh /etc/masakarimonitors/processmonitor.conf /etc/masakarimonitors/proc.list & sudo masakari-hostmonitor.sh /etc/masakarimonitors/hostmonitor.conf & sudo /usr/bin/python ./instancemonitor.py & However the instancemonitor.py starts and exits ... and does not appear to start any process(es) ... with no error messages and no log file. Is this the correct way to startup masakari instance monitoring ? Greg. My Masakari setup in Devstack sudo useradd -s /bin/bash -d /opt/stack -m stack echo "stack ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/stack sudo su - stack git clone https://github.com/openstack-dev/devstack cd devstack local.conf file: [[local|localrc]] ADMIN_PASSWORD=admin DATABASE_PASSWORD=admin RABBIT_PASSWORD=admin SERVICE_PASSWORD=admin # setup Neutron services disable_service n-net enable_service q-svc enable_service q-agt enable_service q-dhcp enable_service q-l3 enable_service q-meta # ceilometer enable_plugin ceilometer https://git.openstack.org/openstack/ceilometer enable_plugin aodh https://git.openstack.org/openstack/aodh # heat enable_plugin heat https://git.openstack.org/openstack/heat # vitrage enable_plugin vitrage https://git.openstack.org/openstack/vitrage enable_plugin vitrage-dashboard https://git.openstack.org/openstack/vitrage-dashboard # masakari enable_plugin masakari git://git.openstack.org/openstack/masakari ./stack.sh __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master
Hi Dinesh, Thanks for the info. 'recovery_method' (choose from 'auto', 'reserved_host', 'auto_priority', 'rh_priority') What should be put for Same as the segment host What should be put as , it seems it accepts any arbitrary values. Any more pointers. Much appreciated. Thank you. Louie Kwan From: Bhor, Dinesh [dinesh.b...@nttdata.com] Sent: Wednesday, December 13, 2017 11:22 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master Hi Greg, Looks like you don’t have “devstack-masakari” host registered in Masakari database. Currently operator have to add all the hosts from failover-segment manually to Masakari database. You can use below command: Register failover-segment to Masakari database: openstack segment create Register hosts under the created segment to Masakari database: openstack segment host create There is a work in progress on auto compute node registration. Please refer: https://blueprints.launchpad.net/masakari-monitors/+spec/auto-compute-node-registration Thank you, Dinesh Bhor From: Waines, Greg Sent: 13 December 2017 19:26:32 To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master ok ... i changed all the domain related attributes in /etc/masakarimonitors/masakarimonitors.conf to be set to ‘default’ . I seemed to get further, but now get the following error when instancemonitor tries to send a notification: Bad Request (HTTP 400) (Request-ID: req-556c6c4d-0e12-414e-ac8c-8ef3a61d4864), Host with name devstack-masakari could not be found. however: - devstack-masakari is in the /etc/hosts - nova hypervisor-list shows devstack-masakari as a hypervisor hostname - i am running both hostmonitor, processmonitor and instancemonitor any ideas ? see details below, Greg. 2017-12-13 13:50:34.353 7637 INFO masakarimonitors.instancemonitor.libvirt_handler.callback [-] Libvirt Event: type=VM, hostname=devstack-masakari, uuid=6ae0b09b-3e93-4f0c-b81b-fa140636f267, time=2017-12-13 13:50:34.353037, event_id=LIFECYCLE, detail=STOPPED_DESTROYED) 2017-12-13 13:50:34.353 7637 INFO masakarimonitors.ha.masakari [-] Send a notification. {'notification': {'hostname': 'devstack-masakari', 'type': 'VM', 'payload': {'instance_uuid': '6ae0b09b-3e93-4f0c-b81b-fa140636f267', 'vir_domain_event': 'STOPPED_DESTROYED', 'event': 'LIFECYCLE'}, 'generated_time': datetime.datetime(2017, 12, 13, 13, 50, 34, 353037)}} 2017-12-13 13:50:34.461 7637 WARNING masakarimonitors.ha.masakari [-] Retry sending a notification. (HttpException: Bad Request (HTTP 400) (Request-ID: req-556c6c4d-0e12-414e-ac8c-8ef3a61d4864), Host with name devstack-masakari could not be found.): HttpException: HttpException: Bad Request (HTTP 400) (Request-ID: req-556c6c4d-0e12-414e-ac8c-8ef3a61d4864), Host with name devstack-masakari could not be found. ^C2017-12-13 13:50:42.462 7637 INFO oslo_service.service [-] Caught SIGINT signal, instantaneous exiting root@devstack-masakari:~# root@devstack-masakari:~# ping devstack-masakari PING localhost (127.0.0.1) 56(84) bytes of data. 64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.049 ms 64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.022 ms ^C --- localhost ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 999ms rtt min/avg/max/mdev = 0.022/0.035/0.049/0.014 ms root@devstack-masakari:~# stack@devstack-masakari:~$ nova hypervisor-list +--+-+---+-+ | ID | Hypervisor hostname | State | Status | +--+-+---+-+ | 5fb1b09b-e5f5-465a-828a-2101135ff700 | devstack-masakari | up| enabled | +--+-+---+-+ stack@devstack-masakari:~$ From: Greg Waines Reply-To: "openstack-dev@lists.openstack.org" Date: Wednesday, December 13, 2017 at 8:17 AM To: "openstack-dev@lists.openstack.org" Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master So i believe the error is: HttpException: HttpException: Expecting to find domain in project. I have attached the /etc/masakarimonitors/masakarimonitors.conf file and the /etc/masakari/masakari.conf file . See the domain related attributes in each file below. Is the Default vs default causing this problem
Re: [openstack-dev] [masakari]problems starting up masakari instance monitoring in devstack @ master
Hello Greg, Please use masakarimonitors.conf instead of hostmonitor.conf and processmonitor.conf. You can generate it by "tox -egenconfig". hostmonitor.conf and processmonitor.conf are used for monitors implemented by shell script. masakarimonitors.conf is a configuration file for monitors implemented by python that you installed. And, we are preparing setting guides. Please see it if you are good. masakari: https://review.openstack.org/#/c/489570/ masakari-monitors: https://review.openstack.org/#/c/489095/ Best regards, On 2017/12/06 22:48, Waines, Greg wrote: I am just getting started working with masakari. I am working on master. I have setup Masakari in Devstack (see details at end of email) ... which starts up masakari-engine and masakari-api processes. I have git cloned the masakari-monitors and started them up (roughly) following the instructions at https://github.com/openstack/masakari-monitors . Specifically: # install & startup monitors cd git clone https://github.com/openstack/masakari-monitors.git cd masakari-monitors sudo python setup.py install cd sudo mkdir /etc/masakarimonitors sudo cp ~/masakari-monitors/etc/masakarimonitors/hostmonitor.conf.sample /etc/masakarimonitors/hostmonitor.conf sudo cp ~/masakari-monitors/etc/masakarimonitors/processmonitor.conf.sample /etc/masakarimonitors/processmonitor.conf sudo cp ~/masakari-monitors/etc/masakarimonitors/proc.list.sample /etc/masakarimonitors/proc.list cd ~/masakari-monitors/masakarimonitors/cmd sudo masakari-processmonitor.sh /etc/masakarimonitors/processmonitor.conf /etc/masakarimonitors/proc.list & sudo masakari-hostmonitor.sh /etc/masakarimonitors/hostmonitor.conf & sudo /usr/bin/python ./instancemonitor.py & However the instancemonitor.py starts and exits ... and does not appear to start any process(es) ... with no error messages and no log file. Is this the correct way to startup masakari instance monitoring ? Greg. My Masakari setup in Devstack sudo useradd -s /bin/bash -d /opt/stack -m stack echo "stack ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/stack sudo su - stack git clone https://github.com/openstack-dev/devstack cd devstack local.conf file: [[local|localrc]] ADMIN_PASSWORD=admin DATABASE_PASSWORD=admin RABBIT_PASSWORD=admin SERVICE_PASSWORD=admin # setup Neutron services disable_service n-net enable_service q-svc enable_service q-agt enable_service q-dhcp enable_service q-l3 enable_service q-meta # ceilometer enable_plugin ceilometer https://git.openstack.org/openstack/ceilometer enable_plugin aodh https://git.openstack.org/openstack/aodh # heat enable_plugin heat https://git.openstack.org/openstack/heat # vitrage enable_plugin vitrage https://git.openstack.org/openstack/vitrage enable_plugin vitrage-dashboard https://git.openstack.org/openstack/vitrage-dashboard # masakari enable_plugin masakari git://git.openstack.org/openstack/masakari ./stack.sh __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- _/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/ Rikimaru Honjo E-mail:honjo.rikim...@po.ntt-tx.co.jp __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master
Hello Greg, I forgot to tell you. Please use process_list.yaml instead of proc.list.sample. On 2017/12/07 14:03, Rikimaru Honjo wrote: Hello Greg, Please use masakarimonitors.conf instead of hostmonitor.conf and processmonitor.conf. You can generate it by "tox -egenconfig". hostmonitor.conf and processmonitor.conf are used for monitors implemented by shell script. masakarimonitors.conf is a configuration file for monitors implemented by python that you installed. And, we are preparing setting guides. Please see it if you are good. masakari: https://review.openstack.org/#/c/489570/ masakari-monitors: https://review.openstack.org/#/c/489095/ Best regards, On 2017/12/06 22:48, Waines, Greg wrote: I am just getting started working with masakari. I am working on master. I have setup Masakari in Devstack (see details at end of email) ... which starts up masakari-engine and masakari-api processes. I have git cloned the masakari-monitors and started them up (roughly) following the instructions at https://github.com/openstack/masakari-monitors . Specifically: # install & startup monitors cd git clone https://github.com/openstack/masakari-monitors.git cd masakari-monitors sudo python setup.py install cd sudo mkdir /etc/masakarimonitors sudo cp ~/masakari-monitors/etc/masakarimonitors/hostmonitor.conf.sample /etc/masakarimonitors/hostmonitor.conf sudo cp ~/masakari-monitors/etc/masakarimonitors/processmonitor.conf.sample /etc/masakarimonitors/processmonitor.conf sudo cp ~/masakari-monitors/etc/masakarimonitors/proc.list.sample /etc/masakarimonitors/proc.list cd ~/masakari-monitors/masakarimonitors/cmd sudo masakari-processmonitor.sh /etc/masakarimonitors/processmonitor.conf /etc/masakarimonitors/proc.list & sudo masakari-hostmonitor.sh /etc/masakarimonitors/hostmonitor.conf & sudo /usr/bin/python ./instancemonitor.py & However the instancemonitor.py starts and exits ... and does not appear to start any process(es) ... with no error messages and no log file. Is this the correct way to startup masakari instance monitoring ? Greg. My Masakari setup in Devstack sudo useradd -s /bin/bash -d /opt/stack -m stack echo "stack ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/stack sudo su - stack git clone https://github.com/openstack-dev/devstack cd devstack local.conf file: [[local|localrc]] ADMIN_PASSWORD=admin DATABASE_PASSWORD=admin RABBIT_PASSWORD=admin SERVICE_PASSWORD=admin # setup Neutron services disable_service n-net enable_service q-svc enable_service q-agt enable_service q-dhcp enable_service q-l3 enable_service q-meta # ceilometer enable_plugin ceilometer https://git.openstack.org/openstack/ceilometer enable_plugin aodh https://git.openstack.org/openstack/aodh # heat enable_plugin heat https://git.openstack.org/openstack/heat # vitrage enable_plugin vitrage https://git.openstack.org/openstack/vitrage enable_plugin vitrage-dashboard https://git.openstack.org/openstack/vitrage-dashboard # masakari enable_plugin masakari git://git.openstack.org/openstack/masakari ./stack.sh __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- _/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/ Rikimaru Honjo E-mail:honjo.rikim...@po.ntt-tx.co.jp __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master
thanks Honjo, I’ll try this out now. I’m assuming that it is ok to start ONLY the instance-monitor if that’s all i want to test. Is that correct ? In that case, following the instructions you pointed me at, I would do the following: · setup devstack oi.e. with ‘enable_plugin masakari git://git.openstack.org/openstack/masakari’ · · ( don’t need corosync and pacemaker ) · · install & startup client ocd ogit clone https://github.com/openstack/python-masakariclient.git ocd python-masakariclient osudo python setup.py build osudo python setup.py install · · install & startup instance monitor ocd ogit clone https://github.com/openstack/masakari-monitors.git osudo mkdir /etc/masakarimonitors ocd masakari-monitors osudo python setup.py build osudo python setup.py install o osudo masakari-instancemonitor & Is this correct ? Greg. From: Rikimaru Honjo Reply-To: "openstack-dev@lists.openstack.org" Date: Thursday, December 7, 2017 at 12:24 AM To: "openstack-dev@lists.openstack.org" Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master Hello Greg, I forgot to tell you. Please use process_list.yaml instead of proc.list.sample. On 2017/12/07 14:03, Rikimaru Honjo wrote: Hello Greg, Please use masakarimonitors.conf instead of hostmonitor.conf and processmonitor.conf. You can generate it by "tox -egenconfig". hostmonitor.conf and processmonitor.conf are used for monitors implemented by shell script. masakarimonitors.conf is a configuration file for monitors implemented by python that you installed. And, we are preparing setting guides. Please see it if you are good. masakari: https://review.openstack.org/#/c/489570/ masakari-monitors: https://review.openstack.org/#/c/489095/ Best regards, On 2017/12/06 22:48, Waines, Greg wrote: I am just getting started working with masakari. I am working on master. I have setup Masakari in Devstack (see details at end of email) ... which starts up masakari-engine and masakari-api processes. I have git cloned the masakari-monitors and started them up (roughly) following the instructions at https://github.com/openstack/masakari-monitors . Specifically: # install & startup monitors cd git clone https://github.com/openstack/masakari-monitors.git cd masakari-monitors sudo python setup.py install cd sudo mkdir /etc/masakarimonitors sudo cp ~/masakari-monitors/etc/masakarimonitors/hostmonitor.conf.sample /etc/masakarimonitors/hostmonitor.conf sudo cp ~/masakari-monitors/etc/masakarimonitors/processmonitor.conf.sample /etc/masakarimonitors/processmonitor.conf sudo cp ~/masakari-monitors/etc/masakarimonitors/proc.list.sample /etc/masakarimonitors/proc.list cd ~/masakari-monitors/masakarimonitors/cmd sudo masakari-processmonitor.sh /etc/masakarimonitors/processmonitor.conf /etc/masakarimonitors/proc.list & sudo masakari-hostmonitor.sh /etc/masakarimonitors/hostmonitor.conf & sudo /usr/bin/python ./instancemonitor.py & However the instancemonitor.py starts and exits ... and does not appear to start any process(es) ... with no error messages and no log file. Is this the correct way to startup masakari instance monitoring ? Greg. My Masakari setup in Devstack sudo useradd -s /bin/bash -d /opt/stack -m stack echo "stack ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/stack sudo su - stack git clone https://github.com/openstack-dev/devstack cd devstack local.conf file: [[local|localrc]] ADMIN_PASSWORD=admin DATABASE_PASSWORD=admin RABBIT_PASSWORD=admin SERVICE_PASSWORD=admin # setup Neutron services disable_service n-net enable_service q-svc enable_service q-agt enable_service q-dhcp enable_service q-l3 enable_service q-meta # ceilometer enable_plugin ceilometer https://git.openstack.org/openstack/ceilometer enable_plugin aodh https://git.openstack.org/openstack/aodh # heat enable_plugin heat https://git.openstack.org/openstack/heat # vitrage enable_plugin vitrage https://git.openstack.org/openstack/vitrage enable_plugin vitrage-dashboard https://git.openstack.org/openstack/vitrage-dashboard # masakari enable_plugin masakari git://git.openstack.org/openstack/masakari ./stack.sh __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org<mailto:openstack-dev-requ...@lists.openstack.org>?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- _/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/ Rikimaru Honjo E-mail:honjo.rikim...@po.ntt-tx.co.jp<mailto:honjo.rikim...@po.ntt-tx.co.jp> __ OpenStack Development Mailing List (
Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master
Honjo, Assuming the masakari instance monitor is up and running, What is the actual USE CASE / TEST CASE for demonstrating instance monitoring ? e.g. is it ? · launch a VM · ps -ef | fgrep qemu// to find the qemu process containing the running VM · kill -9 · · Logs ? owill there be any logs from instance monitor that it detected a failure ? · · instance-monitor will report VM failure to masakari-engine · masakari-engine oLogs ? § will there be any logs from masakari-engine that it received notification of failure ? othen will masakari-engine make the appropriate NOVA calls to recover the VM ? let me know if this is correct, Greg. From: Greg Waines Reply-To: "openstack-dev@lists.openstack.org" Date: Monday, December 11, 2017 at 7:17 AM To: "openstack-dev@lists.openstack.org" Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master thanks Honjo, I’ll try this out now. I’m assuming that it is ok to start ONLY the instance-monitor if that’s all i want to test. Is that correct ? In that case, following the instructions you pointed me at, I would do the following: · setup devstack o i.e. with ‘enable_plugin masakari git://git.openstack.org/openstack/masakari’ · · ( don’t need corosync and pacemaker ) · · install & startup client o cd o git clone https://github.com/openstack/python-masakariclient.git o cd python-masakariclient o sudo python setup.py build o sudo python setup.py install · · install & startup instance monitor o cd o git clone https://github.com/openstack/masakari-monitors.git o sudo mkdir /etc/masakarimonitors o cd masakari-monitors o sudo python setup.py build o sudo python setup.py install o o sudo masakari-instancemonitor & Is this correct ? Greg. From: Rikimaru Honjo Reply-To: "openstack-dev@lists.openstack.org" Date: Thursday, December 7, 2017 at 12:24 AM To: "openstack-dev@lists.openstack.org" Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master Hello Greg, I forgot to tell you. Please use process_list.yaml instead of proc.list.sample. On 2017/12/07 14:03, Rikimaru Honjo wrote: Hello Greg, Please use masakarimonitors.conf instead of hostmonitor.conf and processmonitor.conf. You can generate it by "tox -egenconfig". hostmonitor.conf and processmonitor.conf are used for monitors implemented by shell script. masakarimonitors.conf is a configuration file for monitors implemented by python that you installed. And, we are preparing setting guides. Please see it if you are good. masakari: https://review.openstack.org/#/c/489570/ masakari-monitors: https://review.openstack.org/#/c/489095/ Best regards, On 2017/12/06 22:48, Waines, Greg wrote: I am just getting started working with masakari. I am working on master. I have setup Masakari in Devstack (see details at end of email) ... which starts up masakari-engine and masakari-api processes. I have git cloned the masakari-monitors and started them up (roughly) following the instructions at https://github.com/openstack/masakari-monitors . Specifically: # install & startup monitors cd git clone https://github.com/openstack/masakari-monitors.git cd masakari-monitors sudo python setup.py install cd sudo mkdir /etc/masakarimonitors sudo cp ~/masakari-monitors/etc/masakarimonitors/hostmonitor.conf.sample /etc/masakarimonitors/hostmonitor.conf sudo cp ~/masakari-monitors/etc/masakarimonitors/processmonitor.conf.sample /etc/masakarimonitors/processmonitor.conf sudo cp ~/masakari-monitors/etc/masakarimonitors/proc.list.sample /etc/masakarimonitors/proc.list cd ~/masakari-monitors/masakarimonitors/cmd sudo masakari-processmonitor.sh /etc/masakarimonitors/processmonitor.conf /etc/masakarimonitors/proc.list & sudo masakari-hostmonitor.sh /etc/masakarimonitors/hostmonitor.conf & sudo /usr/bin/python ./instancemonitor.py & However the instancemonitor.py starts and exits ... and does not appear to start any process(es) ... with no error messages and no log file. Is this the correct way to startup masakari instance monitoring ? Greg. My Masakari setup in Devstack sudo useradd -s /bin/bash -d /opt/stack -m stack echo "stack ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/stack sudo su - stack git clone https://github.com/openstack-dev/devstack cd devstack local.conf file: [[local|localrc]] ADMIN_PASSWORD=admin DATABASE_PASSWORD=admin RABBIT_PASSWORD=admin SERVICE_PASSWORD=admin # setup Neutron services disable_service n-net enable_service q-svc enable_service q-agt enable_service q-dhcp enable_service q-l3 enable_service q-meta # ceilometer enable_plugin ceilometer https://git.openstack.org/openstack/cei
Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master
Actually, i’m guessing this is wrong because masakari-instancemonitor fails to start when i use these instructions. stack@devstack-masakari:~/masakari-monitors$ sudo masakari-instancemonitor & [1] 22959 stack@devstack-masakari:~/masakari-monitors$ Traceback (most recent call last): File "/usr/local/bin/masakari-instancemonitor", line 10, in sys.exit(main()) File "/usr/local/lib/python2.7/dist-packages/masakarimonitors/cmd/instancemonitor.py", line 31, in main config.parse_args(sys.argv) File "/usr/local/lib/python2.7/dist-packages/masakarimonitors/config.py", line 32, in parse_args default_config_files=default_config_files) File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2495, in __call__ self._check_required_opts() File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 3134, in _check_required_opts raise RequiredOptError(opt.name, group) oslo_config.cfg.RequiredOptError: value required for option auth-url in group [api] [1]+ Exit 1 sudo masakari-instancemonitor Greg. From: Greg Waines Reply-To: "openstack-dev@lists.openstack.org" Date: Monday, December 11, 2017 at 7:17 AM To: "openstack-dev@lists.openstack.org" Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master thanks Honjo, I’ll try this out now. I’m assuming that it is ok to start ONLY the instance-monitor if that’s all i want to test. Is that correct ? In that case, following the instructions you pointed me at, I would do the following: · setup devstack o i.e. with ‘enable_plugin masakari git://git.openstack.org/openstack/masakari’ · · ( don’t need corosync and pacemaker ) · · install & startup client o cd o git clone https://github.com/openstack/python-masakariclient.git o cd python-masakariclient o sudo python setup.py build o sudo python setup.py install · · install & startup instance monitor o cd o git clone https://github.com/openstack/masakari-monitors.git o sudo mkdir /etc/masakarimonitors o cd masakari-monitors o sudo python setup.py build o sudo python setup.py install o o sudo masakari-instancemonitor & Is this correct ? Greg. From: Rikimaru Honjo Reply-To: "openstack-dev@lists.openstack.org" Date: Thursday, December 7, 2017 at 12:24 AM To: "openstack-dev@lists.openstack.org" Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master Hello Greg, I forgot to tell you. Please use process_list.yaml instead of proc.list.sample. On 2017/12/07 14:03, Rikimaru Honjo wrote: Hello Greg, Please use masakarimonitors.conf instead of hostmonitor.conf and processmonitor.conf. You can generate it by "tox -egenconfig". hostmonitor.conf and processmonitor.conf are used for monitors implemented by shell script. masakarimonitors.conf is a configuration file for monitors implemented by python that you installed. And, we are preparing setting guides. Please see it if you are good. masakari: https://review.openstack.org/#/c/489570/ masakari-monitors: https://review.openstack.org/#/c/489095/ Best regards, On 2017/12/06 22:48, Waines, Greg wrote: I am just getting started working with masakari. I am working on master. I have setup Masakari in Devstack (see details at end of email) ... which starts up masakari-engine and masakari-api processes. I have git cloned the masakari-monitors and started them up (roughly) following the instructions at https://github.com/openstack/masakari-monitors . Specifically: # install & startup monitors cd git clone https://github.com/openstack/masakari-monitors.git cd masakari-monitors sudo python setup.py install cd sudo mkdir /etc/masakarimonitors sudo cp ~/masakari-monitors/etc/masakarimonitors/hostmonitor.conf.sample /etc/masakarimonitors/hostmonitor.conf sudo cp ~/masakari-monitors/etc/masakarimonitors/processmonitor.conf.sample /etc/masakarimonitors/processmonitor.conf sudo cp ~/masakari-monitors/etc/masakarimonitors/proc.list.sample /etc/masakarimonitors/proc.list cd ~/masakari-monitors/masakarimonitors/cmd sudo masakari-processmonitor.sh /etc/masakarimonitors/processmonitor.conf /etc/masakarimonitors/proc.list & sudo masakari-hostmonitor.sh /etc/masakarimonitors/hostmonitor.conf & sudo /usr/bin/python ./instancemonitor.py & However the instancemonitor.py starts and exits ... and does not appear to start any process(es) ... with no error messages and no log file. Is this the correct way to startup masakari instance monitoring ? Greg. My Masakari setup in Devstack sudo useradd -s /bin/bash -d /opt/stack -m stack echo "stack ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/stack sudo su - stack git clone https://github.com/openstack-dev/de
Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master
Went back and started up corosync and pacemaker and now masakari-instancemonitor starts successfully. stack@devstack-masakari:~/masakari-monitors$ sudo masakari-instancemonitor & [1] 25973 stack@devstack-masakari:~/masakari-monitors$ 2017-12-11 12:47:49.483 25974 INFO masakarimonitors.service [-] Starting masakarimonitors-instancemonitor stack@devstack-masakari:~/masakari-monitors$ I don’t see any log file in /var/log for masakari-instancemonitor or masakari-engine ? All processes seem to be up and running fine: stack@devstack-masakari:~/masakari-monitors$ ps -ef | fgrep masakari stack11625 1 0 12:29 ?00:00:10 /usr/bin/python /usr/local/bin/masakari-api --config-file=/etc/masakari/masakari.conf --debug stack11778 1 0 12:29 ?00:00:02 /usr/bin/python /usr/local/bin/masakari-engine --config-file=/etc/masakari/masakari.conf --debug stack12005 11625 0 12:29 ?00:00:00 /usr/bin/python /usr/local/bin/masakari-api --config-file=/etc/masakari/masakari.conf --debug stack12006 11625 0 12:29 ?00:00:00 /usr/bin/python /usr/local/bin/masakari-api --config-file=/etc/masakari/masakari.conf --debug root 19336 1 0 12:19 ?00:00:05 /opt/stack/bin/etcd --name devstack-masakari --data-dir /opt/stack/data/etcd --initial-cluster-state new --initial-cluster-token etcd-cluster-01 --initial-cluster devstack-masakari=http://10.10.10.7:2380 --initial-advertise-peer-urls http://10.10.10.7:2380 --advertise-client-urls http://10.10.10.7:2379 --listen-peer-urls http://0.0.0.0:2380 --listen-client-urls http://10.10.10.7:2379 root 25973 25760 0 12:47 pts/000:00:00 sudo masakari-instancemonitor root 25974 25973 0 12:47 pts/000:00:00 /usr/bin/python /usr/local/bin/masakari-instancemonitor stack26123 25760 0 12:50 pts/000:00:00 grep -F --color=auto masakari rabbitmq 26477 26348 0 12:06 ?00:00:15 /usr/lib/erlang/erts-7.3/bin/beam.smp -W w -A 64 -P 1048576 -K true -B i -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.5.7/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit@devstack-masakari -boot start_sasl -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rab...@devstack-masakari.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rab...@devstack-masakari-sasl.log"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.5.7/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@devstack-masakari-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@devstack-masakari" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672 stack@devstack-masakari:~/masakari-monitors$ Greg. From: Greg Waines Reply-To: "openstack-dev@lists.openstack.org" Date: Monday, December 11, 2017 at 7:38 AM To: "openstack-dev@lists.openstack.org" Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master Actually, i’m guessing this is wrong because masakari-instancemonitor fails to start when i use these instructions. stack@devstack-masakari:~/masakari-monitors$ sudo masakari-instancemonitor & [1] 22959 stack@devstack-masakari:~/masakari-monitors$ Traceback (most recent call last): File "/usr/local/bin/masakari-instancemonitor", line 10, in sys.exit(main()) File "/usr/local/lib/python2.7/dist-packages/masakarimonitors/cmd/instancemonitor.py", line 31, in main config.parse_args(sys.argv) File "/usr/local/lib/python2.7/dist-packages/masakarimonitors/config.py", line 32, in parse_args default_config_files=default_config_files) File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2495, in __call__ self._check_required_opts() File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 3134, in _check_required_opts raise RequiredOptError(opt.name, group) oslo_config.cfg.RequiredOptError: value required for option auth-url in group [api] [1]+ Exit 1 sudo masakari-instancemonitor Greg. From: Greg Waines Reply-To: "openstack-dev@lists.openstack.org" Date: Monday, December 11, 2017 at 7:17 AM To: "openstack-dev@lists.openstack.org" Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master thanks Honjo, I’ll try this out now. I’m assuming that it is ok to start ONLY the instance-monitor if that’s all i want to test. Is that correct ? In that case, following the instructions you pointed me at,
Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master
.536 25974 ERROR masakarimonitors.ha.masakari return super(Session, self).request(*args, **kwargs) 2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 573, in request 2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari auth_headers = self.get_auth_headers(auth) 2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 900, in get_auth_headers 2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari return auth.get_headers(self, **kwargs) 2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/plugin.py", line 95, in get_headers 2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari token = self.get_token(session) 2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 88, in get_token 2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari return self.get_access(session).auth_token 2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access 2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari self.auth_ref = self.get_auth_ref(session) 2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py", line 198, in get_auth_ref 2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari return self._plugin.get_auth_ref(session, **kwargs) 2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/v3/base.py", line 165, in get_auth_ref 2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari authenticated=False, log=False, **rkwargs) 2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 848, in post 2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari return self.request(url, 'POST', **kwargs) 2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 66, in map_exceptions_wrapper 2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari raise exceptions.from_exception(e) 2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari HttpException: HttpException: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-3640dd4f-b9d9-4a10-ae37-43a0e202393f), Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. 2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari stack@devstack-masakari:~/devstack$ Greg. From: Greg Waines Reply-To: "openstack-dev@lists.openstack.org" Date: Monday, December 11, 2017 at 7:52 AM To: "openstack-dev@lists.openstack.org" Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master Went back and started up corosync and pacemaker and now masakari-instancemonitor starts successfully. stack@devstack-masakari:~/masakari-monitors$ sudo masakari-instancemonitor & [1] 25973 stack@devstack-masakari:~/masakari-monitors$ 2017-12-11 12:47:49.483 25974 INFO masakarimonitors.service [-] Starting masakarimonitors-instancemonitor stack@devstack-masakari:~/masakari-monitors$ I don’t see any log file in /var/log for masakari-instancemonitor or masakari-engine ? All processes seem to be up and running fine: stack@devstack-masakari:~/masakari-monitors$ ps -ef | fgrep masakari stack11625 1 0 12:29 ?00:00:10 /usr/bin/python /usr/local/bin/masakari-api --config-file=/etc/masakari/masakari.conf --debug stack11778 1 0 12:29 ?00:00:02 /usr/bin/python /usr/local/bin/masakari-engine --config-file=/etc/masakari/masakari.conf --debug stack12005 11625 0 12:29 ?00:00:00 /usr/bin/python /usr/local/bin/masakari-api --config-file=/etc/masakari/masakari.conf --debug stack12006 11625 0 12:29 ?00:00:00 /usr/bin/python /usr/local/bin/masakari-api --config-file=/etc/masakari/masakari.conf --debug root 19336 1 0 12:19 ?00:00:05 /opt/stack/bin/etcd --name devstack-masakari --data-dir /opt/stack/data/etcd --initial-cluster-state new --initial-cluster-token etcd-clus
Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master
Any thoughts on what i don’t have setup correctly wrt the masakari client from the errors below ? Greg. From: Greg Waines Reply-To: "openstack-dev@lists.openstack.org" Date: Monday, December 11, 2017 at 8:05 AM To: "openstack-dev@lists.openstack.org" Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master I started up a VM and then deleted the VM ... and got some errors that make me think i don’t have the masakari client setup correctly. Any ideas ? Greg. i.e. stack@devstack-masakari:~/devstack$ nova list +--+-+++-+-+ | ID | Name| Status | Task State | Power State | Networks| +--+-+++-+-+ | de8922c6-450d-4e2e-954d-ee4bd05ab909 | vm-1-cirros | ACTIVE | - | Running | private=10.0.0.10, fd1a:8f71:1a96:0:f816:3eff:fe5e:f315 | +--+-+++-+-+ stack@devstack-masakari:~/devstack$ stack@devstack-masakari:~/devstack$ nova delete vm-1-cirros Request to delete server vm-1-cirros has been accepted. stack@devstack-masakari:~/devstack$ 2017-12-11 12:58:28.319 25974 INFO masakarimonitors.instancemonitor.libvirt_handler.callback [-] Libvirt Event: type=VM, hostname=devstack-masakari, uuid=de8922c6-450d-4e2e-954d-ee4bd05ab909, time=2017-12-11 12:58:28.318781, event_id=LIFECYCLE, detail=STOPPED_DESTROYED) 2017-12-11 12:58:28.319 25974 INFO masakarimonitors.ha.masakari [-] Send a notification. {'notification': {'hostname': 'devstack-masakari', 'type': 'VM', 'payload': {'instance_uuid': 'de8922c6-450d-4e2e-954d-ee4bd05ab909', 'vir_domain_event': 'STOPPED_DESTROYED', 'event': 'LIFECYCLE'}, 'generated_time': datetime.datetime(2017, 12, 11, 12, 58, 28, 318781)}} 2017-12-11 12:58:28.351 25974 WARNING masakarimonitors.ha.masakari [-] Retry sending a notification. (HttpException: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-feaa30f9-fbb2-4259-8851-488ff7ab82f9), Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error.): HttpException: HttpException: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-feaa30f9-fbb2-4259-8851-488ff7ab82f9), Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. ... several times ... and then eventually 2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari Traceback (most recent call last): 2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/masakarimonitors/ha/masakari.py", line 91, in send_notification 2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari payload=event['notification']['payload']) 2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/masakariclient/sdk/ha/v1/_proxy.py", line 65, in create_notification 2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari return self._create(_notification.Notification, **attrs) 2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/openstack/proxy2.py", line 194, in _create 2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari return res.create(self._session) 2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/openstack/resource2.py", line 588, in create 2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari json=request.body, headers=request.headers) 2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 848, in post 2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari return self.request(url, 'POST', **kwargs) 2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File &qu
Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master
ok ... i changed all the domain related attributes in /etc/masakarimonitors/masakarimonitors.conf to be set to ‘default’ . I seemed to get further, but now get the following error when instancemonitor tries to send a notification: Bad Request (HTTP 400) (Request-ID: req-556c6c4d-0e12-414e-ac8c-8ef3a61d4864), Host with name devstack-masakari could not be found. however: - devstack-masakari is in the /etc/hosts - nova hypervisor-list shows devstack-masakari as a hypervisor hostname - i am running both hostmonitor, processmonitor and instancemonitor any ideas ? see details below, Greg. 2017-12-13 13:50:34.353 7637 INFO masakarimonitors.instancemonitor.libvirt_handler.callback [-] Libvirt Event: type=VM, hostname=devstack-masakari, uuid=6ae0b09b-3e93-4f0c-b81b-fa140636f267, time=2017-12-13 13:50:34.353037, event_id=LIFECYCLE, detail=STOPPED_DESTROYED) 2017-12-13 13:50:34.353 7637 INFO masakarimonitors.ha.masakari [-] Send a notification. {'notification': {'hostname': 'devstack-masakari', 'type': 'VM', 'payload': {'instance_uuid': '6ae0b09b-3e93-4f0c-b81b-fa140636f267', 'vir_domain_event': 'STOPPED_DESTROYED', 'event': 'LIFECYCLE'}, 'generated_time': datetime.datetime(2017, 12, 13, 13, 50, 34, 353037)}} 2017-12-13 13:50:34.461 7637 WARNING masakarimonitors.ha.masakari [-] Retry sending a notification. (HttpException: Bad Request (HTTP 400) (Request-ID: req-556c6c4d-0e12-414e-ac8c-8ef3a61d4864), Host with name devstack-masakari could not be found.): HttpException: HttpException: Bad Request (HTTP 400) (Request-ID: req-556c6c4d-0e12-414e-ac8c-8ef3a61d4864), Host with name devstack-masakari could not be found. ^C2017-12-13 13:50:42.462 7637 INFO oslo_service.service [-] Caught SIGINT signal, instantaneous exiting root@devstack-masakari:~# root@devstack-masakari:~# ping devstack-masakari PING localhost (127.0.0.1) 56(84) bytes of data. 64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.049 ms 64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.022 ms ^C --- localhost ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 999ms rtt min/avg/max/mdev = 0.022/0.035/0.049/0.014 ms root@devstack-masakari:~# stack@devstack-masakari:~$ nova hypervisor-list +--+-+---+-+ | ID | Hypervisor hostname | State | Status | +--+-+---+-+ | 5fb1b09b-e5f5-465a-828a-2101135ff700 | devstack-masakari | up| enabled | +--+-+---+-+ stack@devstack-masakari:~$ From: Greg Waines Reply-To: "openstack-dev@lists.openstack.org" Date: Wednesday, December 13, 2017 at 8:17 AM To: "openstack-dev@lists.openstack.org" Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master So i believe the error is: HttpException: HttpException: Expecting to find domain in project. I have attached the /etc/masakarimonitors/masakarimonitors.conf file and the /etc/masakari/masakari.conf file . See the domain related attributes in each file below. Is the Default vs default causing this problem ? Are there other domain related attributes that should be set to default ? stack@devstack-masakari:~$ fgrep -i domain /etc/masakari/masakari.conf project_domain_name = Default user_domain_name = Default stack@devstack-masakari:~$ fgrep -i domain /etc/masakarimonitors/masakarimonitors.conf #logging_user_identity_format = %(user)s %(tenant)s %(domain)s %(user_domain)s %(project_domain)s # Domain ID to scope to (string value) #domain_id = # Domain name to scope to (string value) #domain_name = # Domain ID containing project (string value) #project_domain_id = # Domain name containing project (string value) #project_domain_name = project_domain_name = default # Optional domain ID to use with v3 and v2 parameters. It will be used for both # the user and project domain in v3 and ignored in v2 authentication. (string #default_domain_id = # Optional domain name to use with v3 API and v2 parameters. It will be used for # both the user and project domain in v3 and ignored in v2 authentication. #default_domain_name = # User's domain id (string value) #user_domain_id = # User's domain name (string value) #user_domain_name = # Indicate whether this resource may be shared with the domain received in the stack@devstack-masakari:~$ Greg. From: Greg Waines Reply-To: "openstack-dev@lists.openstack.org" Date: Tuesday, December 12, 2017 at 7:19 AM To: "openstack-dev@lists.openstack.org" Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master Any thoughts on what i don’t hav
Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master
Hi Greg, Looks like you don’t have “devstack-masakari” host registered in Masakari database. Currently operator have to add all the hosts from failover-segment manually to Masakari database. You can use below command: Register failover-segment to Masakari database: openstack segment create Register hosts under the created segment to Masakari database: openstack segment host create There is a work in progress on auto compute node registration. Please refer: https://blueprints.launchpad.net/masakari-monitors/+spec/auto-compute-node-registration Thank you, Dinesh Bhor From: Waines, Greg Sent: 13 December 2017 19:26:32 To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master ok ... i changed all the domain related attributes in /etc/masakarimonitors/masakarimonitors.conf to be set to ‘default’ . I seemed to get further, but now get the following error when instancemonitor tries to send a notification: Bad Request (HTTP 400) (Request-ID: req-556c6c4d-0e12-414e-ac8c-8ef3a61d4864), Host with name devstack-masakari could not be found. however: - devstack-masakari is in the /etc/hosts - nova hypervisor-list shows devstack-masakari as a hypervisor hostname - i am running both hostmonitor, processmonitor and instancemonitor any ideas ? see details below, Greg. 2017-12-13 13:50:34.353 7637 INFO masakarimonitors.instancemonitor.libvirt_handler.callback [-] Libvirt Event: type=VM, hostname=devstack-masakari, uuid=6ae0b09b-3e93-4f0c-b81b-fa140636f267, time=2017-12-13 13:50:34.353037, event_id=LIFECYCLE, detail=STOPPED_DESTROYED) 2017-12-13 13:50:34.353 7637 INFO masakarimonitors.ha.masakari [-] Send a notification. {'notification': {'hostname': 'devstack-masakari', 'type': 'VM', 'payload': {'instance_uuid': '6ae0b09b-3e93-4f0c-b81b-fa140636f267', 'vir_domain_event': 'STOPPED_DESTROYED', 'event': 'LIFECYCLE'}, 'generated_time': datetime.datetime(2017, 12, 13, 13, 50, 34, 353037)}} 2017-12-13 13:50:34.461 7637 WARNING masakarimonitors.ha.masakari [-] Retry sending a notification. (HttpException: Bad Request (HTTP 400) (Request-ID: req-556c6c4d-0e12-414e-ac8c-8ef3a61d4864), Host with name devstack-masakari could not be found.): HttpException: HttpException: Bad Request (HTTP 400) (Request-ID: req-556c6c4d-0e12-414e-ac8c-8ef3a61d4864), Host with name devstack-masakari could not be found. ^C2017-12-13 13:50:42.462 7637 INFO oslo_service.service [-] Caught SIGINT signal, instantaneous exiting root@devstack-masakari:~# root@devstack-masakari:~# ping devstack-masakari PING localhost (127.0.0.1) 56(84) bytes of data. 64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.049 ms 64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.022 ms ^C --- localhost ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 999ms rtt min/avg/max/mdev = 0.022/0.035/0.049/0.014 ms root@devstack-masakari:~# stack@devstack-masakari:~$ nova hypervisor-list +--+-+---+-+ | ID | Hypervisor hostname | State | Status | +--+-+---+-+ | 5fb1b09b-e5f5-465a-828a-2101135ff700 | devstack-masakari | up| enabled | +--+-+---+-+ stack@devstack-masakari:~$ From: Greg Waines Reply-To: "openstack-dev@lists.openstack.org" Date: Wednesday, December 13, 2017 at 8:17 AM To: "openstack-dev@lists.openstack.org" Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master So i believe the error is: HttpException: HttpException: Expecting to find domain in project. I have attached the /etc/masakarimonitors/masakarimonitors.conf file and the /etc/masakari/masakari.conf file . See the domain related attributes in each file below. Is the Default vs default causing this problem ? Are there other domain related attributes that should be set to default ? stack@devstack-masakari:~$ fgrep -i domain /etc/masakari/masakari.conf project_domain_name = Default user_domain_name = Default stack@devstack-masakari:~$ fgrep -i domain /etc/masakarimonitors/masakarimonitors.conf #logging_user_identity_format = %(user)s %(tenant)s %(domain)s %(user_domain)s %(project_domain)s # Domain ID to scope to (string value) #domain_id = # Domain name to scope to (string value) #domain_name = # Domain ID containing project (string value) #project_domain_id = # Domain name containing project (string value) #project_domain_name = project_d