I started up a VM and then deleted the VM ... and got some errors that make me
think i don’t have the masakari client setup correctly.
Any ideas ?
Greg.
i.e.
stack@devstack-masakari:~/devstack$ nova list
+--------------------------------------+-------------+--------+------------+-------------+---------------------------------------------------------+
| ID | Name | Status | Task State |
Power State | Networks |
+--------------------------------------+-------------+--------+------------+-------------+---------------------------------------------------------+
| de8922c6-450d-4e2e-954d-ee4bd05ab909 | vm-1-cirros | ACTIVE | - |
Running | private=10.0.0.10, fd1a:8f71:1a96:0:f816:3eff:fe5e:f315 |
+--------------------------------------+-------------+--------+------------+-------------+---------------------------------------------------------+
stack@devstack-masakari:~/devstack$
stack@devstack-masakari:~/devstack$ nova delete vm-1-cirros
Request to delete server vm-1-cirros has been accepted.
stack@devstack-masakari:~/devstack$ 2017-12-11 12:58:28.319 25974 INFO
masakarimonitors.instancemonitor.libvirt_handler.callback [-] Libvirt Event:
type=VM, hostname=devstack-masakari, uuid=de8922c6-450d-4e2e-954d-ee4bd05ab909,
time=2017-12-11 12:58:28.318781, event_id=LIFECYCLE, detail=STOPPED_DESTROYED)
2017-12-11 12:58:28.319 25974 INFO masakarimonitors.ha.masakari [-] Send a
notification. {'notification': {'hostname': 'devstack-masakari', 'type': 'VM',
'payload': {'instance_uuid': 'de8922c6-450d-4e2e-954d-ee4bd05ab909',
'vir_domain_event': 'STOPPED_DESTROYED', 'event': 'LIFECYCLE'},
'generated_time': datetime.datetime(2017, 12, 11, 12, 58, 28, 318781)}}
2017-12-11 12:58:28.351 25974 WARNING masakarimonitors.ha.masakari [-] Retry
sending a notification. (HttpException: Expecting to find domain in project.
The server could not comply with the request since it is either malformed or
otherwise incorrect. The client is assumed to be in error. (HTTP 400)
(Request-ID: req-feaa30f9-fbb2-4259-8851-488ff7ab82f9), Expecting to find
domain in project. The server could not comply with the request since it is
either malformed or otherwise incorrect. The client is assumed to be in
error.): HttpException: HttpException: Expecting to find domain in project. The
server could not comply with the request since it is either malformed or
otherwise incorrect. The client is assumed to be in error. (HTTP 400)
(Request-ID: req-feaa30f9-fbb2-4259-8851-488ff7ab82f9), Expecting to find
domain in project. The server could not comply with the request since it is
either malformed or otherwise incorrect. The client is assumed to be in error.
... several times ... and then eventually
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari Traceback
(most recent call last):
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File
"/usr/local/lib/python2.7/dist-packages/masakarimonitors/ha/masakari.py", line
91, in send_notification
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari
payload=event['notification']['payload'])
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File
"/usr/local/lib/python2.7/dist-packages/masakariclient/sdk/ha/v1/_proxy.py",
line 65, in create_notification
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari return
self._create(_notification.Notification, **attrs)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File
"/usr/local/lib/python2.7/dist-packages/openstack/proxy2.py", line 194, in
_create
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari return
res.create(self._session)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File
"/usr/local/lib/python2.7/dist-packages/openstack/resource2.py", line 588, in
create
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari
json=request.body, headers=request.headers)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File
"/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 848, in
post
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari return
self.request(url, 'POST', **kwargs)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File
"/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 64, in
map_exceptions_wrapper
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari return
func(*args, **kwargs)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File
"/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 352, in
request
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari return
super(Session, self).request(*args, **kwargs)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File
"/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 573, in
request
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari
auth_headers = self.get_auth_headers(auth)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File
"/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 900, in
get_auth_headers
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari return
auth.get_headers(self, **kwargs)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File
"/usr/local/lib/python2.7/dist-packages/keystoneauth1/plugin.py", line 95, in
get_headers
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari token =
self.get_token(session)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File
"/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line
88, in get_token
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari return
self.get_access(session).auth_token
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File
"/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line
134, in get_access
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari
self.auth_ref = self.get_auth_ref(session)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File
"/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py",
line 198, in get_auth_ref
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari return
self._plugin.get_auth_ref(session, **kwargs)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File
"/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/v3/base.py",
line 165, in get_auth_ref
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari
authenticated=False, log=False, **rkwargs)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File
"/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 848, in
post
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari return
self.request(url, 'POST', **kwargs)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File
"/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 66, in
map_exceptions_wrapper
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari raise
exceptions.from_exception(e)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari HttpException:
HttpException: Expecting to find domain in project. The server could not comply
with the request since it is either malformed or otherwise incorrect. The
client is assumed to be in error. (HTTP 400) (Request-ID:
req-3640dd4f-b9d9-4a10-ae37-43a0e202393f), Expecting to find domain in project.
The server could not comply with the request since it is either malformed or
otherwise incorrect. The client is assumed to be in error.
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari
stack@devstack-masakari:~/devstack$
Greg.
From: Greg Waines <[email protected]>
Reply-To: "[email protected]"
<[email protected]>
Date: Monday, December 11, 2017 at 7:52 AM
To: "[email protected]" <[email protected]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance
monitoring in devstack @ master
Went back and started up corosync and pacemaker and now
masakari-instancemonitor starts successfully.
stack@devstack-masakari:~/masakari-monitors$ sudo masakari-instancemonitor &
[1] 25973
stack@devstack-masakari:~/masakari-monitors$ 2017-12-11 12:47:49.483 25974 INFO
masakarimonitors.service [-] Starting masakarimonitors-instancemonitor
stack@devstack-masakari:~/masakari-monitors$
I don’t see any log file in /var/log for masakari-instancemonitor or
masakari-engine ?
All processes seem to be up and running fine:
stack@devstack-masakari:~/masakari-monitors$ ps -ef | fgrep masakari
stack 11625 1 0 12:29 ? 00:00:10 /usr/bin/python
/usr/local/bin/masakari-api --config-file=/etc/masakari/masakari.conf --debug
stack 11778 1 0 12:29 ? 00:00:02 /usr/bin/python
/usr/local/bin/masakari-engine --config-file=/etc/masakari/masakari.conf --debug
stack 12005 11625 0 12:29 ? 00:00:00 /usr/bin/python
/usr/local/bin/masakari-api --config-file=/etc/masakari/masakari.conf --debug
stack 12006 11625 0 12:29 ? 00:00:00 /usr/bin/python
/usr/local/bin/masakari-api --config-file=/etc/masakari/masakari.conf --debug
root 19336 1 0 12:19 ? 00:00:05 /opt/stack/bin/etcd --name
devstack-masakari --data-dir /opt/stack/data/etcd --initial-cluster-state new
--initial-cluster-token etcd-cluster-01 --initial-cluster
devstack-masakari=http://10.10.10.7:2380 --initial-advertise-peer-urls
http://10.10.10.7:2380 --advertise-client-urls http://10.10.10.7:2379
--listen-peer-urls http://0.0.0.0:2380 --listen-client-urls
http://10.10.10.7:2379
root 25973 25760 0 12:47 pts/0 00:00:00 sudo masakari-instancemonitor
root 25974 25973 0 12:47 pts/0 00:00:00 /usr/bin/python
/usr/local/bin/masakari-instancemonitor
stack 26123 25760 0 12:50 pts/0 00:00:00 grep -F --color=auto masakari
rabbitmq 26477 26348 0 12:06 ? 00:00:15
/usr/lib/erlang/erts-7.3/bin/beam.smp -W w -A 64 -P 1048576 -K true -B i --
-root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa
/usr/lib/rabbitmq/lib/rabbitmq_server-3.5.7/sbin/../ebin -noshell -noinput -s
rabbit boot -sname rabbit@devstack-masakari -boot start_sasl -kernel
inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl
sasl_error_logger false -rabbit error_logger
{file,"/var/log/rabbitmq/[email protected]"} -rabbit
sasl_error_logger {file,"/var/log/rabbitmq/[email protected]"}
-rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit
plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.5.7/sbin/../plugins"
-rabbit plugins_expand_dir
"/var/lib/rabbitmq/mnesia/rabbit@devstack-masakari-plugins-expand" -os_mon
start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false
-mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@devstack-masakari" -kernel
inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672
stack@devstack-masakari:~/masakari-monitors$
Greg.
From: Greg Waines <[email protected]>
Reply-To: "[email protected]"
<[email protected]>
Date: Monday, December 11, 2017 at 7:38 AM
To: "[email protected]" <[email protected]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance
monitoring in devstack @ master
Actually, i’m guessing this is wrong because masakari-instancemonitor fails to
start when i use these instructions.
stack@devstack-masakari:~/masakari-monitors$ sudo masakari-instancemonitor &
[1] 22959
stack@devstack-masakari:~/masakari-monitors$ Traceback (most recent call last):
File "/usr/local/bin/masakari-instancemonitor", line 10, in <module>
sys.exit(main())
File
"/usr/local/lib/python2.7/dist-packages/masakarimonitors/cmd/instancemonitor.py",
line 31, in main
config.parse_args(sys.argv)
File "/usr/local/lib/python2.7/dist-packages/masakarimonitors/config.py",
line 32, in parse_args
default_config_files=default_config_files)
File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2495,
in __call__
self._check_required_opts()
File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 3134,
in _check_required_opts
raise RequiredOptError(opt.name, group)
oslo_config.cfg.RequiredOptError: value required for option auth-url in group
[api]
[1]+ Exit 1 sudo masakari-instancemonitor
Greg.
From: Greg Waines <[email protected]>
Reply-To: "[email protected]"
<[email protected]>
Date: Monday, December 11, 2017 at 7:17 AM
To: "[email protected]" <[email protected]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance
monitoring in devstack @ master
thanks Honjo,
I’ll try this out now.
I’m assuming that it is ok to start ONLY the instance-monitor if that’s all i
want to test.
Is that correct ?
In that case, following the instructions you pointed me at,
I would do the following:
· setup devstack
o i.e. with ‘enable_plugin masakari
git://git.openstack.org/openstack/masakari’
·
· ( don’t need corosync and pacemaker )
·
· install & startup client
o cd
o git clone https://github.com/openstack/python-masakariclient.git
o cd python-masakariclient
o sudo python setup.py build
o sudo python setup.py install
·
· install & startup instance monitor
o cd
o git clone https://github.com/openstack/masakari-monitors.git
o sudo mkdir /etc/masakarimonitors
o cd masakari-monitors
o sudo python setup.py build
o sudo python setup.py install
o
o sudo masakari-instancemonitor &
Is this correct ?
Greg.
From: Rikimaru Honjo <[email protected]>
Reply-To: "[email protected]"
<[email protected]>
Date: Thursday, December 7, 2017 at 12:24 AM
To: "[email protected]" <[email protected]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance
monitoring in devstack @ master
Hello Greg,
I forgot to tell you.
Please use process_list.yaml instead of proc.list.sample.
On 2017/12/07 14:03, Rikimaru Honjo wrote:
Hello Greg,
Please use masakarimonitors.conf instead of hostmonitor.conf and
processmonitor.conf.
You can generate it by "tox -egenconfig".
hostmonitor.conf and processmonitor.conf are used for monitors implemented by
shell script.
masakarimonitors.conf is a configuration file for monitors implemented by
python that you installed.
And, we are preparing setting guides.
Please see it if you are good.
masakari:
https://review.openstack.org/#/c/489570/
masakari-monitors:
https://review.openstack.org/#/c/489095/
Best regards,
On 2017/12/06 22:48, Waines, Greg wrote:
I am just getting started working with masakari.
I am working on master.
I have setup Masakari in Devstack (see details at end of email) ... which
starts up masakari-engine and masakari-api processes.
I have git cloned the masakari-monitors and started them up (roughly) following
the instructions at https://github.com/openstack/masakari-monitors .
Specifically:
# install & startup monitors
cd
git clone https://github.com/openstack/masakari-monitors.git
cd masakari-monitors
sudo python setup.py install
cd
sudo mkdir /etc/masakarimonitors
sudo cp ~/masakari-monitors/etc/masakarimonitors/hostmonitor.conf.sample
/etc/masakarimonitors/hostmonitor.conf
sudo cp ~/masakari-monitors/etc/masakarimonitors/processmonitor.conf.sample
/etc/masakarimonitors/processmonitor.conf
sudo cp ~/masakari-monitors/etc/masakarimonitors/proc.list.sample
/etc/masakarimonitors/proc.list
cd ~/masakari-monitors/masakarimonitors/cmd
sudo masakari-processmonitor.sh /etc/masakarimonitors/processmonitor.conf
/etc/masakarimonitors/proc.list &
sudo masakari-hostmonitor.sh /etc/masakarimonitors/hostmonitor.conf &
sudo /usr/bin/python ./instancemonitor.py &
However the instancemonitor.py starts and exits ... and does not appear to
start any process(es) ... with no error messages and no log file.
Is this the correct way to startup masakari instance monitoring ?
Greg.
My Masakari setup in Devstack
sudo useradd -s /bin/bash -d /opt/stack -m stack
echo "stack ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/stack
sudo su - stack
git clone https://github.com/openstack-dev/devstack
cd devstack
local.conf file:
[[local|localrc]]
ADMIN_PASSWORD=admin
DATABASE_PASSWORD=admin
RABBIT_PASSWORD=admin
SERVICE_PASSWORD=admin
# setup Neutron services
disable_service n-net
enable_service q-svc
enable_service q-agt
enable_service q-dhcp
enable_service q-l3
enable_service q-meta
# ceilometer
enable_plugin ceilometer https://git.openstack.org/openstack/ceilometer
enable_plugin aodh https://git.openstack.org/openstack/aodh
# heat
enable_plugin heat https://git.openstack.org/openstack/heat
# vitrage
enable_plugin vitrage https://git.openstack.org/openstack/vitrage
enable_plugin vitrage-dashboard
https://git.openstack.org/openstack/vitrage-dashboard
# masakari
enable_plugin masakari git://git.openstack.org/openstack/masakari
./stack.sh
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
[email protected]<mailto:[email protected]>?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
--
_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/
Rikimaru Honjo
E-mail:[email protected]<mailto:[email protected]>
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
[email protected]<mailto:[email protected]>?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [email protected]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
