Hi Guys, I have a new openstack cloud that I am unable to provision any instances on. When provisioning via horizon the state of the instance gets stuck for a long time at the Scheduling state and then the Build state and never completes. I see the following time outs in the logs:
Not really sure what is going on and would really appreciate any help on pointing me to some direction: -- /var/log/nova/nova-compute-json.log:{"domain": null, "color": "", "@timestamp": "2017-05-05T18:20:47.717Z", "isotime": "2017-05-05T18:20:47.717599+00:00", "resource_uuid": null, "message": "Failed to revert task state for instance. Error: Timed out waiting for a reply to message ID 9ec8ad37f9504e549855a6331757da8a", "read_only": false, "user_id": "07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a", "user_identity": "07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a 206a2db080e942a292df932654e528e5 - - -", "instance": "[instance: 6bd76ba0-ac26-4954-9613-e39c3bb081be] ", "version": "unknown", "read_deleted": "no", "logger": "nova.compute.manager", "project_id": "206a2db080e942a292df932654e528e5", "type": "nova", "project_name": "gredadmin", "tags": "nova-compute", "timestamp": "2017-05-05T18:11:44.031234", "remote_address": "10.39.75.15", "quota_class": null, "host": "gredhos2-cp1-comp0006-mgmt", "is_admin": true, "user": "07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a", "service_catalog": [{"endpoints": [{"adminURL": "https://10.39.75.51:8776/v1/206a2db080e942a292df932654e528e5", "region": "region1", "internalURL": "https://10.39.75.51:8776/v1/206a2db080e942a292df932654e528e5", "publicURL": "https://gredcloud2.gene.com:8776/v1/206a2db080e942a292df932654e528e5"}], "type": "volume", "name": "cinder"}, {"endpoints": [{"adminURL": "https://10.39.75.51:8776/v2/206a2db080e942a292df932654e528e5", "region": "region1", "internalURL": "https://10.39.75.51:8776/v2/206a2db080e942a292df932654e528e5", "publicURL": "https://gredcloud2.gene.com:8776/v2/206a2db080e942a292df932654e528e5"}], "type": "volumev2", "name": "cinderv2"}, {"endpoints": [{"adminURL": "https://10.39.75.51:9311", "region": "region1", "internalURL": "https://10.39.75.51:9311", "publicURL": "https://gredcloud2.gene.com:9311"}], "type": "key-manager", "name": "barbican"}], "path": "/opt/stack/venv/nova-20170214T094710Z/lib/python2.7/site-packages/nova/compute/manager.py", "extra_keys": ["instance", "project", "version"], "instance_lock_checked": false, "tenant": "206a2db080e942a292df932654e528e5", "resource": "", "show_deleted": false, "roles": ["admin"], "user_name": "simran", "project": "unknown", "project_domain": null, "request_id": "req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06", "user_domain": null, "@version": "1", "levelname": "WARNING"} /var/log/nova/nova-compute.log:2017-05-05 18:18:47.698 1943 DEBUG oslo_concurrency.lockutils [req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06 07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a 206a2db080e942a292df932654e528e5] Lock "6bd76ba0-ac26-4954-9613-e39c3bb081be" acquired by "nova.compute.manager._locked_do_build_and_run_instance" :: waited 0.000s inner /opt/stack/venv/nova-20170214T094710Z/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270 -- Here are all the errors I see regarding this instances: -- $ cat /var/tmp/sim.out | grep -i error /var/log/kronos/kronos-api.log:2017-05-05 18:21:02.366 31593 DEBUG kafka.client [-] Request 926437 to BrokerMetadata(nodeId=0, host='10.39.75.15', port=9092): [ProduceRequest(topic='logs', partition=22, messages=[Message(magic=0, attributes=0, key='1494008462366', value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008462,"log":{"message":"May 5 18:21:01 gredhos2-cp1-c1-m3-mgmt drive-audit: Devices found: [{\'block_device\': \'\\/dev\\/sdb1\', \'major\': \'8\', \'mount_point\': \'\\/srv\\/node\\/disk0\', \'kernel_device\': \'sdb1\', \'minor\': \'17\'}, {\'block_device\': \'\\/dev\\/sdc1\', \'major\': \'8\', \'mount_point\': \'\\/srv\\/node\\/disk1\', \'kernel_device\': \'sdc1\', \'minor\': \'33\'}]","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:21:01.835Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\/var\\/log\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'), Message(magic=0, attributes=0, key='1494008462366', value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008462,"log":{"message":"May 5 18:21:01 gredhos2-cp1-c1-m3-mgmt drive-audit: Errors found: {}","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:21:01.835Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\/var\\/log\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'), Message(magic=0, attributes=0, key='1494008462366', value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008462,"log":{"message":"May 5 18:21:01 gredhos2-cp1-c1-m3-mgmt drive-audit: No drives were unmounted","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:21:01.835Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\/var\\/log\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'), Message(magic=0, attributes=0, key='1494008462366', value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008462,"log":{"message":"May 5 18:21:01 gredhos2-cp1-c1-m3-mgmt ceilometer-agent-notification[27729]: curl -i -X POST -H \'X-Auth-User: ceilometer\' -H \'X-Auth-Token: *****\' -H \'X-Auth-Key: *****\' -H \'Accept: application\\/json\' -H \'User-Agent: python-monascaclient\' -H \'Content-Type: application\\/json\' -H \'X-Auth-Url: https:\\/\\/gredhos2-cp1-vip-KEY-API-mgmt:5000\\/v3\' -d \'[{\\"timestamp\\": 1494008424708, \\"dimensions\\": {\\"user_id\\": \\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\", \\"cloud_name\\": \\"gredcloud2\\", \\"region\\": \\"None\\", \\"resource_id\\": \\"6bd76ba0-ac26-4954-9613-e39c3bb081be\\", \\"control_plane\\": \\"control-plane-1\\", \\"cluster\\": \\"cluster1\\", \\"datasource\\": \\"ceilometer\\", \\"project_id\\": \\"206a2db080e942a292df932654e528e5\\", \\"type\\": \\"gauge\\", \\"unit\\": \\"instance\\", \\"source\\": \\"openstack\\"}, \\"name\\": \\"instance\\", \\"value\\": 1, \\"value_meta\\": {\\"audit_period_beginning\\": \\"2017-05-05T18:00:00.000000\\", \\"root_gb\\": 20, \\"event_type\\": \\"compute.instance.update\\", \\"disk_gb\\": 20, \\"availability_zone\\": \\"nova\\", \\"instance_flavor_id\\": \\"2\\", \\"ephemeral_gb\\": 0, \\"vcpus\\": 1, \\"memory_mb\\": 2048, \\"host\\": \\"compute.gredhos2-cp1-c1-m3-mgmt\\", \\"state\\": \\"building\\", \\"audit_period_ending\\": \\"2017-05-05T18:20:14.370128\\", \\"created_at\\": \\"2017-05-05 18:11:58+00:00\\", \\"deleted_at\\": \\"\\", \\"launched_at\\": \\"\\", \\"image_ref_url\\": \\"http:\\/\\/10.39.75.17:9292\\/images\\/f4bff4e4-466b-4482-a1ed-2a3ac5131ff2\\"}}, {\\"timestamp\\": 1494008424708, \\"dimensions\\": {\\"user_id\\": \\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\", \\"cloud_name\\": \\"gredcloud2\\", \\"region\\": \\"None\\", \\"resource_id\\": \\"6bd76ba0-ac26-4954-9613-e39c3bb081be\\", \\"control_plane\\": \\"control-plane-1\\", \\"cluster\\": \\"cluster1\\", \\"datasource\\": \\"ceilometer\\", \\"project_id\\": \\"206a2db080e942a292df932654e528e5\\", \\"type\\": \\"gauge\\", \\"unit\\": \\"vcpu\\", \\"source\\": \\"openstack\\"}, \\"name\\": \\"vcpus\\", \\"value\\": 1, \\"value_meta\\": {\\"audit_period_beginning\\": \\"2017-05-05T18:00:00.000000\\", \\"root_gb\\": 20, \\"event_type\\": \\"compute.instance.update\\", \\"disk_gb\\": 20, \\"availability_zone\\": \\"nova\\", \\"instance_flavor_id\\": \\"2\\", \\"ephemeral_gb\\": 0, \\"vcpus\\": 1, \\"memory_mb\\": 2048, \\"host\\": \\"compute.gredhos2-cp1-c1-m3-mgmt\\", \\"state\\": \\"building\\", \\"audit_period_ending\\": \\"2017-05-05T","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:21:01.835Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\/var\\/log\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}')])] _send_broker_aware_request /opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207 /var/log/kronos/wsgi.log:[Fri May 05 18:21:02.366960 2017] [wsgi:error] [pid 31593:tid 139973023282944] 2017-05-05 18:21:02.366 31593 DEBUG kafka.client [-] Request 926437 to BrokerMetadata(nodeId=0, host='10.39.75.15', port=9092): [ProduceRequest(topic='logs', partition=22, messages=[Message(magic=0, attributes=0, key='1494008462366', value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008462,"log":{"message":"May 5 18:21:01 gredhos2-cp1-c1-m3-mgmt drive-audit: Devices found: [{\\'block_device\\': \\'\\\\/dev\\\\/sdb1\\', \\'major\\': \\'8\\', \\'mount_point\\': \\'\\\\/srv\\\\/node\\\\/disk0\\', \\'kernel_device\\': \\'sdb1\\', \\'minor\\': \\'17\\'}, {\\'block_device\\': \\'\\\\/dev\\\\/sdc1\\', \\'major\\': \\'8\\', \\'mount_point\\': \\'\\\\/srv\\\\/node\\\\/disk1\\', \\'kernel_device\\': \\'sdc1\\', \\'minor\\': \\'33\\'}]","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:21:01.835Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'), Message(magic=0, attributes=0, key='1494008462366', value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008462,"log":{"message":"May 5 18:21:01 gredhos2-cp1-c1-m3-mgmt drive-audit: Errors found: {}","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:21:01.835Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'), Message(magic=0, attributes=0, key='1494008462366', value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008462,"log":{"message":"May 5 18:21:01 gredhos2-cp1-c1-m3-mgmt drive-audit: No drives were unmounted","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:21:01.835Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'), Message(magic=0, attributes=0, key='1494008462366', value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008462,"log":{"message":"May 5 18:21:01 gredhos2-cp1-c1-m3-mgmt ceilometer-agent-notification[27729]: curl -i -X POST -H \\'X-Auth-User: ceilometer\\' -H \\'X-Auth-Token: *****\\' -H \\'X-Auth-Key: *****\\' -H \\'Accept: application\\\\/json\\' -H \\'User-Agent: python-monascaclient\\' -H \\'Content-Type: application\\\\/json\\' -H \\'X-Auth-Url: https:\\\\/\\\\/gredhos2-cp1-vip-KEY-API-mgmt:5000\\\\/v3\\' -d \\'[{\\\\"timestamp\\\\": 1494008424708, \\\\"dimensions\\\\": {\\\\"user_id\\\\": \\\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\\\", \\\\"cloud_name\\\\": \\\\"gredcloud2\\\\", \\\\"region\\\\": \\\\"None\\\\", \\\\"resource_id\\\\": \\\\"6bd76ba0-ac26-4954-9613-e39c3bb081be\\\\", \\\\"control_plane\\\\": \\\\"control-plane-1\\\\", \\\\"cluster\\\\": \\\\"cluster1\\\\", \\\\"datasource\\\\": \\\\"ceilometer\\\\", \\\\"project_id\\\\": \\\\"206a2db080e942a292df932654e528e5\\\\", \\\\"type\\\\": \\\\"gauge\\\\", \\\\"unit\\\\": \\\\"instance\\\\", \\\\"source\\\\": \\\\"openstack\\\\"}, \\\\"name\\\\": \\\\"instance\\\\", \\\\"value\\\\": 1, \\\\"value_meta\\\\": {\\\\"audit_period_beginning\\\\": \\\\"2017-05-05T18:00:00.000000\\\\", \\\\"root_gb\\\\": 20, \\\\"event_type\\\\": \\\\"compute.instance.update\\\\", \\\\"disk_gb\\\\": 20, \\\\"availability_zone\\\\": \\\\"nova\\\\", \\\\"instance_flavor_id\\\\": \\\\"2\\\\", \\\\"ephemeral_gb\\\\": 0, \\\\"vcpus\\\\": 1, \\\\"memory_mb\\\\": 2048, \\\\"host\\\\": \\\\"compute.gredhos2-cp1-c1-m3-mgmt\\\\", \\\\"state\\\\": \\\\"building\\\\", \\\\"audit_period_ending\\\\": \\\\"2017-05-05T18:20:14.370128\\\\", \\\\"created_at\\\\": \\\\"2017-05-05 18:11:58+00:00\\\\", \\\\"deleted_at\\\\": \\\\"\\\\", \\\\"launched_at\\\\": \\\\"\\\\", \\\\"image_ref_url\\\\": \\\\"http:\\\\/\\\\/10.39.75.17:9292\\\\/images\\\\/f4bff4e4-466b-4482-a1ed-2a3ac5131ff2\\\\"}}, {\\\\"timestamp\\\\": 1494008424708, \\\\"dimensions\\\\": {\\\\"user_id\\\\": \\\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\\\", \\\\"cloud_name\\\\": \\\\"gredcloud2\\\\", \\\\"region\\\\": \\\\"None\\\\", \\\\"resource_id\\\\": \\\\"6bd76ba0-ac26-4954-9613-e39c3bb081be\\\\", \\\\"control_plane\\\\": \\\\"control-plane-1\\\\", \\\\"cluster\\\\": \\\\"cluster1\\\\", \\\\"datasource\\\\": \\\\"ceilometer\\\\", \\\\"project_id\\\\": \\\\"206a2db080e942a292df932654e528e5\\\\", \\\\"type\\\\": \\\\"gauge\\\\", \\\\"unit\\\\": \\\\"vcpu\\\\", \\\\"source\\\\": \\\\"openstack\\\\"}, \\\\"name\\\\": \\\\"vcpus\\\\", \\\\"value\\\\": 1, \\\\"value_meta\\\\": {\\\\"audit_period_beginning\\\\": \\\\"2017-05-05T18:00:00.000000\\\\", \\\\"root_gb\\\\": 20, \\\\"event_type\\\\": \\\\"compute.instance.update\\\\", \\\\"disk_gb\\\\": 20, \\\\"availability_zone\\\\": \\\\"nova\\\\", \\\\"instance_flavor_id\\\\": \\\\"2\\\\", \\\\"ephemeral_gb\\\\": 0, \\\\"vcpus\\\\": 1, \\\\"memory_mb\\\\": 2048, \\\\"host\\\\": \\\\"compute.gredhos2-cp1-c1-m3-mgmt\\\\", \\\\"state\\\\": \\\\"building\\\\", \\\\"audit_period_ending\\\\": \\\\"2017-05-05T","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:21:01.835Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}')])] _send_broker_aware_request /opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207 /var/log/kronos/wsgi.log:[Fri May 05 18:21:02.380248 2017] [wsgi:error] [pid 31585:tid 139973031675648] 2017-05-05 18:21:02.379 31585 DEBUG kafka.client [-] Request 932067 to BrokerMetadata(nodeId=0, host='10.39.75.15', port=9092): [ProduceRequest(topic='logs', partition=46, messages=[Message(magic=0, attributes=0, key='1494008462379', value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008462,"log":{"message":"May 5 18:21:01 gredhos2-cp1-c1-m3-mgmt ceilometer-agent-notification[27729]: 18:20:14.370128\\\\", \\\\"created_at\\\\": \\\\"2017-05-05 18:11:58+00:00\\\\", \\\\"deleted_at\\\\": \\\\"\\\\", \\\\"launched_at\\\\": \\\\"\\\\", \\\\"image_ref_url\\\\": \\\\"http:\\\\/\\\\/10.39.75.17:9292\\\\/images\\\\/f4bff4e4-466b-4482-a1ed-2a3ac5131ff2\\\\"}}, {\\\\"timestamp\\\\": 1494008424708, \\\\"dimensions\\\\": {\\\\"user_id\\\\": \\\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\\\", \\\\"cloud_name\\\\": \\\\"gredcloud2\\\\", \\\\"region\\\\": \\\\"None\\\\", \\\\"resource_id\\\\": \\\\"6bd76ba0-ac26-4954-9613-e39c3bb081be\\\\", \\\\"control_plane\\\\": \\\\"control-plane-1\\\\", \\\\"cluster\\\\": \\\\"cluster1\\\\", \\\\"datasource\\\\": \\\\"ceilometer\\\\", \\\\"project_id\\\\": \\\\"206a2db080e942a292df932654e528e5\\\\", \\\\"type\\\\": \\\\"gauge\\\\", \\\\"unit\\\\": \\\\"MB\\\\", \\\\"source\\\\": \\\\"openstack\\\\"}, \\\\"name\\\\": \\\\"memory\\\\", \\\\"value\\\\": 2048, \\\\"value_meta\\\\": {\\\\"audit_period_beginning\\\\": \\\\"2017-05-05T18:00:00.000000\\\\", \\\\"root_gb\\\\": 20, \\\\"event_type\\\\": \\\\"compute.instance.update\\\\", \\\\"disk_gb\\\\": 20, \\\\"availability_zone\\\\": \\\\"nova\\\\", \\\\"instance_flavor_id\\\\": \\\\"2\\\\", \\\\"ephemeral_gb\\\\": 0, \\\\"vcpus\\\\": 1, \\\\"memory_mb\\\\": 2048, \\\\"host\\\\": \\\\"compute.gredhos2-cp1-c1-m3-mgmt\\\\", \\\\"state\\\\": \\\\"building\\\\", \\\\"audit_period_ending\\\\": \\\\"2017-05-05T18:20:14.370128\\\\", \\\\"created_at\\\\": \\\\"2017-05-05 18:11:58+00:00\\\\", \\\\"deleted_at\\\\": \\\\"\\\\", \\\\"launched_at\\\\": \\\\"\\\\", \\\\"image_ref_url\\\\": \\\\"http:\\\\/\\\\/10.39.75.17:9292\\\\/images\\\\/f4bff4e4-466b-4482-a1ed-2a3ac5131ff2\\\\"}}, {\\\\"timestamp\\\\": 1494008424708, \\\\"dimensions\\\\": {\\\\"user_id\\\\": \\\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\\\", \\\\"cloud_name\\\\": \\\\"gredcloud2\\\\", \\\\"region\\\\": \\\\"None\\\\", \\\\"resource_id\\\\": \\\\"6bd76ba0-ac26-4954-9613-e39c3bb081be\\\\", \\\\"control_plane\\\\": \\\\"control-plane-1\\\\", \\\\"cluster\\\\": \\\\"cluster1\\\\", \\\\"datasource\\\\": \\\\"ceilometer\\\\", \\\\"project_id\\\\": \\\\"206a2db080e942a292df932654e528e5\\\\", \\\\"type\\\\": \\\\"gauge\\\\", \\\\"unit\\\\": \\\\"GB\\\\", \\\\"source\\\\": \\\\"openstack\\\\"}, \\\\"name\\\\": \\\\"disk.ephemeral.size\\\\", \\\\"value\\\\": 0, \\\\"value_meta\\\\": {\\\\"audit_period_ending\\\\": \\\\"2017-05-05T18:20:14.370128\\\\", \\\\"audit_period_beginning\\\\": \\\\"2017-05-05T18:00:00.000000\\\\", \\\\"event_type\\\\": \\\\"compute.instance.update\\\\", \\\\"availability_zone\\\\": \\\\"nova\\\\"}}, {\\\\"timestamp\\\\": 1494008424708, \\\\"dimensions\\\\": {\\\\"user_id\\\\": \\\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\\\", \\\\"cloud_name\\\\": \\\\"gredcloud2\\\\", \\\\"region\\\\": \\\\"None\\\\", \\\\"resource_id\\\\": \\\\"6bd76ba0-ac26-4954-9613-e39c3bb081be","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:21:01.835Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'), Message(magic=0, attributes=0, key='1494008462379', value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008462,"log":{"message":"May 5 18:21:01 gredhos2-cp1-c1-m3-mgmt ceilometer-agent-notification[27729]: \\\\", \\\\"control_plane\\\\": \\\\"control-plane-1\\\\", \\\\"cluster\\\\": \\\\"cluster1\\\\", \\\\"datasource\\\\": \\\\"ceilometer\\\\", \\\\"project_id\\\\": \\\\"206a2db080e942a292df932654e528e5\\\\", \\\\"type\\\\": \\\\"gauge\\\\", \\\\"unit\\\\": \\\\"GB\\\\", \\\\"source\\\\": \\\\"openstack\\\\"}, \\\\"name\\\\": \\\\"disk.root.size\\\\", \\\\"value\\\\": 20, \\\\"value_meta\\\\": {\\\\"audit_period_ending\\\\": \\\\"2017-05-05T18:20:14.370128\\\\", \\\\"audit_period_beginning\\\\": \\\\"2017-05-05T18:00:00.000000\\\\", \\\\"event_type\\\\": \\\\"compute.instance.update\\\\", \\\\"availability_zone\\\\": \\\\"nova\\\\"}}]\\' https:\\\\/\\\\/gredhos2-cp1-vip-MON-API-mgmt:8070\\\\/v2.0\\\\/metrics","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:21:01.835Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}')])] _send_broker_aware_request /opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207 /var/log/kronos/kronos-api.log:2017-05-05 18:18:47.705 19722 DEBUG kafka.client [-] Request 922321 to BrokerMetadata(nodeId=1, host='10.39.75.16', port=9092): [ProduceRequest(topic='logs', partition=29, messages=[Message(magic=0, attributes=0, key='1494008327705', value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008327,"log":{"message":{"domain":null,"color":"","@timestamp":"2017-05-05T18:18:47.419Z","isotime":"2017-05-05T18:18:47.419068+00:00","resource_uuid":null,"message":"Error from last host: gredhos2-cp1-comp0002-mgmt (node gredhos2-cp1-comp0002-mgmt): [u\'Traceback (most recent call last):\\\\n\', u\' File \\"\\/opt\\/stack\\/venv\\/nova-20170214T094710Z\\/lib\\/python2.7\\/site-packages\\/nova\\/compute\\/manager.py\\", line 1930, in _do_build_and_run_instance\\\\n filter_properties)\\\\n\', u\' File \\"\\/opt\\/stack\\/venv\\/nova-20170214T094710Z\\/lib\\/python2.7\\/site-packages\\/nova\\/compute\\/manager.py\\", line 2120, in _build_and_run_instance\\\\n instance_uuid=instance.uuid, reason=six.text_type(e))\\\\n\', u\'RescheduledException: Build of instance 6bd76ba0-ac26-4954-9613-e39c3bb081be was re-scheduled: Timed out waiting for a reply to message ID b2c5a1fbfe4e48c38954f6ba9bb6e04b\\\\n\']","read_only":false,"user_id":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","user_identity":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a 206a2db080e942a292df932654e528e5 - - -","instance":"[instance: 6bd76ba0-ac26-4954-9613-e39c3bb081be] ","version":"unknown","read_deleted":"no","logger":"nova.scheduler.utils","project_id":"206a2db080e942a292df932654e528e5","type":"nova","project_name":"gredadmin","tags":"nova-conductor","timestamp":"2017-05-05T18:11:44.031234","remote_address":"10.39.75.15","quota_class":null,"host":"gredhos2-cp1-c1-m3-mgmt","is_admin":true,"user":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","service_catalog":[{"endpoints":[{"adminURL":"https:\\/\\/10.39.75.51:8776\\/v1\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\/\\/10.39.75.51:8776\\/v1\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\/\\/gredcloud2.gene.com:8776\\/v1\\/206a2db080e942a292df932654e528e5"}],"type":"volume","name":"cinder"},{"endpoints":[{"adminURL":"https:\\/\\/10.39.75.51:8776\\/v2\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\/\\/10.39.75.51:8776\\/v2\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\/\\/gredcloud2.gene.com:8776\\/v2\\/206a2db080e942a292df932654e528e5"}],"type":"volumev2","name":"cinderv2"},{"endpoints":[{"adminURL":"https:\\/\\/10.39.75.51:9311","region":"region1","internalURL":"https:\\/\\/10.39.75.51:9311","publicURL":"https:\\/\\/gredcloud2.gene.com:9311"}],"type":"key-manager","name":"barbican"}],"path":"\\/opt\\/stack\\/venv\\/nova-20170214T094710Z\\/lib\\/python2.7\\/site-packages\\/nova\\/scheduler\\/utils.py","extra_keys":["instance_uuid","project","version"],"instance_lock_checked":false,"tenant":"206a2db080e942a292df932654e528e5","instance_uuid":"6bd76ba0-ac26-4954-9613-e39c3bb081be","resource":"","show_deleted":false,"roles":["admin"],"user_name":"simran","project":"unknown","project_domain":null,"request_id":"req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06","user_domain":null,"@version":"1","levelname":"ERROR"},"dimensions":{"service":"nova","message_format_source":"json","message_format":"json","type":"nova-api","@timestamp":"2017-05-05T18:18:47.488Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\/var\\/log\\/nova\\/nova-conductor-json.log","@version":"1","cloud_name":"gredcloud2"}}}')])] _send_broker_aware_request /opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207 /var/log/kronos/kronos-api.log:2017-05-05 18:20:47.511 19722 DEBUG kafka.client [-] Request 923735 to BrokerMetadata(nodeId=1, host='10.39.75.16', port=9092): [ProduceRequest(topic='logs', partition=62, messages=[Message(magic=0, attributes=0, key='1494008447511', value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008447,"log":{"message":{"domain":null,"color":"","@timestamp":"2017-05-05T18:20:47.425Z","isotime":"2017-05-05T18:20:47.425718+00:00","resource_uuid":null,"message":"Failed to revert task state for instance. Error: Timed out waiting for a reply to message ID a76c427f58744af7bb512c666812ac98","read_only":false,"user_id":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","user_identity":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a 206a2db080e942a292df932654e528e5 - - -","instance":"[instance: 6bd76ba0-ac26-4954-9613-e39c3bb081be] ","version":"unknown","read_deleted":"no","logger":"nova.compute.manager","project_id":"206a2db080e942a292df932654e528e5","type":"nova","project_name":"gredadmin","tags":"nova-compute","timestamp":"2017-05-05T18:11:44.031234","remote_address":"10.39.75.15","quota_class":null,"host":"gredhos2-cp1-comp0002-mgmt","is_admin":true,"user":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","service_catalog":[{"endpoints":[{"adminURL":"https:\\/\\/10.39.75.51:8776\\/v1\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\/\\/10.39.75.51:8776\\/v1\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\/\\/gredcloud2.gene.com:8776\\/v1\\/206a2db080e942a292df932654e528e5"}],"type":"volume","name":"cinder"},{"endpoints":[{"adminURL":"https:\\/\\/10.39.75.51:8776\\/v2\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\/\\/10.39.75.51:8776\\/v2\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\/\\/gredcloud2.gene.com:8776\\/v2\\/206a2db080e942a292df932654e528e5"}],"type":"volumev2","name":"cinderv2"},{"endpoints":[{"adminURL":"https:\\/\\/10.39.75.51:9311","region":"region1","internalURL":"https:\\/\\/10.39.75.51:9311","publicURL":"https:\\/\\/gredcloud2.gene.com:9311"}],"type":"key-manager","name":"barbican"}],"path":"\\/opt\\/stack\\/venv\\/nova-20170214T094710Z\\/lib\\/python2.7\\/site-packages\\/nova\\/compute\\/manager.py","extra_keys":["instance","project","version"],"instance_lock_checked":false,"tenant":"206a2db080e942a292df932654e528e5","resource":"","show_deleted":false,"roles":["admin"],"user_name":"simran","project":"unknown","project_domain":null,"request_id":"req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06","user_domain":null,"@version":"1","levelname":"WARNING"},"dimensions":{"service":"nova","message_format_source":"json","message_format":"json","type":"nova-compute","@timestamp":"2017-05-05T18:20:47.495Z","control_plane":"control-plane-1","cluster":"compute","host":"gredhos2-cp1-comp0002-mgmt","file":"\\/var\\/log\\/nova\\/nova-compute-json.log","@version":"1","cloud_name":"gredcloud2"}}}')])] _send_broker_aware_request /opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207 /var/log/kronos/wsgi.log:[Fri May 05 18:18:47.705385 2017] [wsgi:error] [pid 19722:tid 140044119824128] 2017-05-05 18:18:47.705 19722 DEBUG kafka.client [-] Request 922321 to BrokerMetadata(nodeId=1, host='10.39.75.16', port=9092): [ProduceRequest(topic='logs', partition=29, messages=[Message(magic=0, attributes=0, key='1494008327705', value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008327,"log":{"message":{"domain":null,"color":"","@timestamp":"2017-05-05T18:18:47.419Z","isotime":"2017-05-05T18:18:47.419068+00:00","resource_uuid":null,"message":"Error from last host: gredhos2-cp1-comp0002-mgmt (node gredhos2-cp1-comp0002-mgmt): [u\\'Traceback (most recent call last):\\\\\\\\n\\', u\\' File \\\\"\\\\/opt\\\\/stack\\\\/venv\\\\/nova-20170214T094710Z\\\\/lib\\\\/python2.7\\\\/site-packages\\\\/nova\\\\/compute\\\\/manager.py\\\\", line 1930, in _do_build_and_run_instance\\\\\\\\n filter_properties)\\\\\\\\n\\', u\\' File \\\\"\\\\/opt\\\\/stack\\\\/venv\\\\/nova-20170214T094710Z\\\\/lib\\\\/python2.7\\\\/site-packages\\\\/nova\\\\/compute\\\\/manager.py\\\\", line 2120, in _build_and_run_instance\\\\\\\\n instance_uuid=instance.uuid, reason=six.text_type(e))\\\\\\\\n\\', u\\'RescheduledException: Build of instance 6bd76ba0-ac26-4954-9613-e39c3bb081be was re-scheduled: Timed out waiting for a reply to message ID b2c5a1fbfe4e48c38954f6ba9bb6e04b\\\\\\\\n\\']","read_only":false,"user_id":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","user_identity":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a 206a2db080e942a292df932654e528e5 - - -","instance":"[instance: 6bd76ba0-ac26-4954-9613-e39c3bb081be] ","version":"unknown","read_deleted":"no","logger":"nova.scheduler.utils","project_id":"206a2db080e942a292df932654e528e5","type":"nova","project_name":"gredadmin","tags":"nova-conductor","timestamp":"2017-05-05T18:11:44.031234","remote_address":"10.39.75.15","quota_class":null,"host":"gredhos2-cp1-c1-m3-mgmt","is_admin":true,"user":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","service_catalog":[{"endpoints":[{"adminURL":"https:\\\\/\\\\/10.39.75.51:8776\\\\/v1\\\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\\\/\\\\/10.39.75.51:8776\\\\/v1\\\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\\\/\\\\/gredcloud2.gene.com:8776\\\\/v1\\\\/206a2db080e942a292df932654e528e5"}],"type":"volume","name":"cinder"},{"endpoints":[{"adminURL":"https:\\\\/\\\\/10.39.75.51:8776\\\\/v2\\\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\\\/\\\\/10.39.75.51:8776\\\\/v2\\\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\\\/\\\\/gredcloud2.gene.com:8776\\\\/v2\\\\/206a2db080e942a292df932654e528e5"}],"type":"volumev2","name":"cinderv2"},{"endpoints":[{"adminURL":"https:\\\\/\\\\/10.39.75.51:9311","region":"region1","internalURL":"https:\\\\/\\\\/10.39.75.51:9311","publicURL":"https:\\\\/\\\\/gredcloud2.gene.com:9311"}],"type":"key-manager","name":"barbican"}],"path":"\\\\/opt\\\\/stack\\\\/venv\\\\/nova-20170214T094710Z\\\\/lib\\\\/python2.7\\\\/site-packages\\\\/nova\\\\/scheduler\\\\/utils.py","extra_keys":["instance_uuid","project","version"],"instance_lock_checked":false,"tenant":"206a2db080e942a292df932654e528e5","instance_uuid":"6bd76ba0-ac26-4954-9613-e39c3bb081be","resource":"","show_deleted":false,"roles":["admin"],"user_name":"simran","project":"unknown","project_domain":null,"request_id":"req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06","user_domain":null,"@version":"1","levelname":"ERROR"},"dimensions":{"service":"nova","message_format_source":"json","message_format":"json","type":"nova-api","@timestamp":"2017-05-05T18:18:47.488Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/nova\\\\/nova-conductor-json.log","@version":"1","cloud_name":"gredcloud2"}}}')])] _send_broker_aware_request /opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207 /var/log/kronos/wsgi.log:[Fri May 05 18:18:51.722268 2017] [wsgi:error] [pid 19713:tid 140044128216832] 2017-05-05 18:18:51.722 19713 DEBUG kafka.client [-] Request 919983 to BrokerMetadata(nodeId=2, host='10.39.75.17', port=9092): [ProduceRequest(topic='logs', partition=126, messages=[Message(magic=0, attributes=0, key='1494008331721', value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008331,"log":{"message":"May 5 18:18:51 gredhos2-cp1-c1-m3-mgmt ceilometer-agent-notification[27729]: curl -i -X POST -H \\'X-Auth-User: ceilometer\\' -H \\'X-Auth-Token: *****\\' -H \\'X-Auth-Key: *****\\' -H \\'Accept: application\\\\/json\\' -H \\'User-Agent: python-monascaclient\\' -H \\'Content-Type: application\\\\/json\\' -H \\'X-Auth-Url: https:\\\\/\\\\/gredhos2-cp1-vip-KEY-API-mgmt:5000\\\\/v3\\' -d \\'[{\\\\"timestamp\\\\": 1494008318345, \\\\"dimensions\\\\": {\\\\"user_id\\\\": \\\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\\\", \\\\"cloud_name\\\\": \\\\"gredcloud2\\\\", \\\\"region\\\\": \\\\"None\\\\", \\\\"resource_id\\\\": \\\\"6bd76ba0-ac26-4954-9613-e39c3bb081be\\\\", \\\\"control_plane\\\\": \\\\"control-plane-1\\\\", \\\\"cluster\\\\": \\\\"cluster1\\\\", \\\\"datasource\\\\": \\\\"ceilometer\\\\", \\\\"project_id\\\\": \\\\"206a2db080e942a292df932654e528e5\\\\", \\\\"type\\\\": \\\\"gauge\\\\", \\\\"unit\\\\": \\\\"instance\\\\", \\\\"source\\\\": \\\\"openstack\\\\"}, \\\\"name\\\\": \\\\"instance\\\\", \\\\"value\\\\": 1, \\\\"value_meta\\\\": {\\\\"audit_period_beginning\\\\": \\\\"2017-05-05T18:00:00.000000\\\\", \\\\"root_gb\\\\": 20, \\\\"event_type\\\\": \\\\"compute.instance.update\\\\", \\\\"disk_gb\\\\": 20, \\\\"availability_zone\\\\": \\\\"nova\\\\", \\\\"instance_flavor_id\\\\": \\\\"2\\\\", \\\\"ephemeral_gb\\\\": 0, \\\\"vcpus\\\\": 1, \\\\"memory_mb\\\\": 2048, \\\\"host\\\\": \\\\"compute.gredhos2-cp1-c1-m3-mgmt\\\\", \\\\"state\\\\": \\\\"building\\\\", \\\\"audit_period_ending\\\\": \\\\"2017-05-05T18:18:38.338459\\\\", \\\\"created_at\\\\": \\\\"2017-05-05 18:11:58+00:00\\\\", \\\\"deleted_at\\\\": \\\\"\\\\", \\\\"launched_at\\\\": \\\\"\\\\", \\\\"image_ref_url\\\\": \\\\"http:\\\\/\\\\/10.39.75.17:9292\\\\/images\\\\/f4bff4e4-466b-4482-a1ed-2a3ac5131ff2\\\\"}}, {\\\\"timestamp\\\\": 1494008318345, \\\\"dimensions\\\\": {\\\\"user_id\\\\": \\\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\\\", \\\\"cloud_name\\\\": \\\\"gredcloud2\\\\", \\\\"region\\\\": \\\\"None\\\\", \\\\"resource_id\\\\": \\\\"6bd76ba0-ac26-4954-9613-e39c3bb081be\\\\", \\\\"control_plane\\\\": \\\\"control-plane-1\\\\", \\\\"cluster\\\\": \\\\"cluster1\\\\", \\\\"datasource\\\\": \\\\"ceilometer\\\\", \\\\"project_id\\\\": \\\\"206a2db080e942a292df932654e528e5\\\\", \\\\"type\\\\": \\\\"gauge\\\\", \\\\"unit\\\\": \\\\"vcpu\\\\", \\\\"source\\\\": \\\\"openstack\\\\"}, \\\\"name\\\\": \\\\"vcpus\\\\", \\\\"value\\\\": 1, \\\\"value_meta\\\\": {\\\\"audit_period_beginning\\\\": \\\\"2017-05-05T18:00:00.000000\\\\", \\\\"root_gb\\\\": 20, \\\\"event_type\\\\": \\\\"compute.instance.update\\\\", \\\\"disk_gb\\\\": 20, \\\\"availability_zone\\\\": \\\\"nova\\\\", \\\\"instance_flavor_id\\\\": \\\\"2\\\\", \\\\"ephemeral_gb\\\\": 0, \\\\"vcpus\\\\": 1, \\\\"memory_mb\\\\": 2048, \\\\"host\\\\": \\\\"compute.gredhos2-cp1-c1-m3-mgmt\\\\", \\\\"state\\\\": \\\\"building\\\\", \\\\"audit_period_ending\\\\": \\\\"2017-05-05T","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:18:51.715Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}')])] _send_broker_aware_request /opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207 /var/log/kronos/wsgi.log:[Fri May 05 18:18:51.738044 2017] [wsgi:error] [pid 19722:tid 140044136609536] 2017-05-05 18:18:51.737 19722 DEBUG kafka.client [-] Request 922344 to BrokerMetadata(nodeId=2, host='10.39.75.17', port=9092): [ProduceRequest(topic='logs', partition=123, messages=[Message(magic=0, attributes=0, key='1494008331737', value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008331,"log":{"message":"May 5 18:18:51 gredhos2-cp1-c1-m3-mgmt ceilometer-agent-notification[27729]: 18:18:38.338459\\\\", \\\\"created_at\\\\": \\\\"2017-05-05 18:11:58+00:00\\\\", \\\\"deleted_at\\\\": \\\\"\\\\", \\\\"launched_at\\\\": \\\\"\\\\", \\\\"image_ref_url\\\\": \\\\"http:\\\\/\\\\/10.39.75.17:9292\\\\/images\\\\/f4bff4e4-466b-4482-a1ed-2a3ac5131ff2\\\\"}}, {\\\\"timestamp\\\\": 1494008318345, \\\\"dimensions\\\\": {\\\\"user_id\\\\": \\\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\\\", \\\\"cloud_name\\\\": \\\\"gredcloud2\\\\", \\\\"region\\\\": \\\\"None\\\\", \\\\"resource_id\\\\": \\\\"6bd76ba0-ac26-4954-9613-e39c3bb081be\\\\", \\\\"control_plane\\\\": \\\\"control-plane-1\\\\", \\\\"cluster\\\\": \\\\"cluster1\\\\", \\\\"datasource\\\\": \\\\"ceilometer\\\\", \\\\"project_id\\\\": \\\\"206a2db080e942a292df932654e528e5\\\\", \\\\"type\\\\": \\\\"gauge\\\\", \\\\"unit\\\\": \\\\"MB\\\\", \\\\"source\\\\": \\\\"openstack\\\\"}, \\\\"name\\\\": \\\\"memory\\\\", \\\\"value\\\\": 2048, \\\\"value_meta\\\\": {\\\\"audit_period_beginning\\\\": \\\\"2017-05-05T18:00:00.000000\\\\", \\\\"root_gb\\\\": 20, \\\\"event_type\\\\": \\\\"compute.instance.update\\\\", \\\\"disk_gb\\\\": 20, \\\\"availability_zone\\\\": \\\\"nova\\\\", \\\\"instance_flavor_id\\\\": \\\\"2\\\\", \\\\"ephemeral_gb\\\\": 0, \\\\"vcpus\\\\": 1, \\\\"memory_mb\\\\": 2048, \\\\"host\\\\": \\\\"compute.gredhos2-cp1-c1-m3-mgmt\\\\", \\\\"state\\\\": \\\\"building\\\\", \\\\"audit_period_ending\\\\": \\\\"2017-05-05T18:18:38.338459\\\\", \\\\"created_at\\\\": \\\\"2017-05-05 18:11:58+00:00\\\\", \\\\"deleted_at\\\\": \\\\"\\\\", \\\\"launched_at\\\\": \\\\"\\\\", \\\\"image_ref_url\\\\": \\\\"http:\\\\/\\\\/10.39.75.17:9292\\\\/images\\\\/f4bff4e4-466b-4482-a1ed-2a3ac5131ff2\\\\"}}, {\\\\"timestamp\\\\": 1494008318345, \\\\"dimensions\\\\": {\\\\"user_id\\\\": \\\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\\\", \\\\"cloud_name\\\\": \\\\"gredcloud2\\\\", \\\\"region\\\\": \\\\"None\\\\", \\\\"resource_id\\\\": \\\\"6bd76ba0-ac26-4954-9613-e39c3bb081be\\\\", \\\\"control_plane\\\\": \\\\"control-plane-1\\\\", \\\\"cluster\\\\": \\\\"cluster1\\\\", \\\\"datasource\\\\": \\\\"ceilometer\\\\", \\\\"project_id\\\\": \\\\"206a2db080e942a292df932654e528e5\\\\", \\\\"type\\\\": \\\\"gauge\\\\", \\\\"unit\\\\": \\\\"GB\\\\", \\\\"source\\\\": \\\\"openstack\\\\"}, \\\\"name\\\\": \\\\"disk.ephemeral.size\\\\", \\\\"value\\\\": 0, \\\\"value_meta\\\\": {\\\\"audit_period_ending\\\\": \\\\"2017-05-05T18:18:38.338459\\\\", \\\\"audit_period_beginning\\\\": \\\\"2017-05-05T18:00:00.000000\\\\", \\\\"event_type\\\\": \\\\"compute.instance.update\\\\", \\\\"availability_zone\\\\": \\\\"nova\\\\"}}, {\\\\"timestamp\\\\": 1494008318345, \\\\"dimensions\\\\": {\\\\"user_id\\\\": \\\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\\\", \\\\"cloud_name\\\\": \\\\"gredcloud2\\\\", \\\\"region\\\\": \\\\"None\\\\", \\\\"resource_id\\\\": \\\\"6bd76ba0-ac26-4954-9613-e39c3bb081be","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:18:51.715Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'), Message(magic=0, attributes=0, key='1494008331737', value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008331,"log":{"message":"May 5 18:18:51 gredhos2-cp1-c1-m3-mgmt ceilometer-agent-notification[27729]: \\\\", \\\\"control_plane\\\\": \\\\"control-plane-1\\\\", \\\\"cluster\\\\": \\\\"cluster1\\\\", \\\\"datasource\\\\": \\\\"ceilometer\\\\", \\\\"project_id\\\\": \\\\"206a2db080e942a292df932654e528e5\\\\", \\\\"type\\\\": \\\\"gauge\\\\", \\\\"unit\\\\": \\\\"GB\\\\", \\\\"source\\\\": \\\\"openstack\\\\"}, \\\\"name\\\\": \\\\"disk.root.size\\\\", \\\\"value\\\\": 20, \\\\"value_meta\\\\": {\\\\"audit_period_ending\\\\": \\\\"2017-05-05T18:18:38.338459\\\\", \\\\"audit_period_beginning\\\\": \\\\"2017-05-05T18:00:00.000000\\\\", \\\\"event_type\\\\": \\\\"compute.instance.update\\\\", \\\\"availability_zone\\\\": \\\\"nova\\\\"}}]\\' https:\\\\/\\\\/gredhos2-cp1-vip-MON-API-mgmt:8070\\\\/v2.0\\\\/metrics","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:18:51.715Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'), Message(magic=0, attributes=0, key='1494008331737', value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008331,"log":{"message":"May 5 18:18:51 gredhos2-cp1-c1-m3-mgmt ovs-vsctl: ovs|00001|vsctl|INFO|Called as \\\\/usr\\\\/bin\\\\/ovs-vsctl --timeout=10 --oneline --format=json -- set Port qr-31af2869-0c other_config:net_uuid=0467e434-3356-45f3-8ca1-71cdc9c2dea6 other_config:segmentation_id=1086 other_config:tag=1 other_config:physical_network=None other_config:network_type=vxlan","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:18:51.715Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'), Message(magic=0, attributes=0, key='1494008331737', value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008331,"log":{"message":"May 5 18:18:51 gredhos2-cp1-c1-m3-mgmt ovs-vsctl: ovs|00001|vsctl|INFO|Called as \\\\/usr\\\\/bin\\\\/ovs-vsctl --timeout=10 --oneline --format=json -- set Port sg-3a472f79-0f other_config:net_uuid=adff4284-025b-4c8f-863c-06e59ec6e5a8 other_config:segmentation_id=1052 other_config:tag=2 other_config:physical_network=None other_config:network_type=vxlan","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:18:51.715Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'), Message(magic=0, attributes=0, key='1494008331737', value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008331,"log":{"message":"May 5 18:18:51 gredhos2-cp1-c1-m3-mgmt ceilometer-agent-notification[27729]: HTTP\\\\/1.1 204 No Content","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:18:51.715Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'), Message(magic=0, attributes=0, key='1494008331737', value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008331,"log":{"message":"May 5 18:18:51 gredhos2-cp1-c1-m3-mgmt ceilometer-agent-notification[27729]: Date: Fri, 05 May 2017 18:18:51 GMT","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:18:51.715Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}')])] _send_broker_aware_request /opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207 /var/log/kronos/wsgi.log:[Fri May 05 18:20:47.511872 2017] [wsgi:error] [pid 19722:tid 140044136609536] 2017-05-05 18:20:47.511 19722 DEBUG kafka.client [-] Request 923735 to BrokerMetadata(nodeId=1, host='10.39.75.16', port=9092): [ProduceRequest(topic='logs', partition=62, messages=[Message(magic=0, attributes=0, key='1494008447511', value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008447,"log":{"message":{"domain":null,"color":"","@timestamp":"2017-05-05T18:20:47.425Z","isotime":"2017-05-05T18:20:47.425718+00:00","resource_uuid":null,"message":"Failed to revert task state for instance. Error: Timed out waiting for a reply to message ID a76c427f58744af7bb512c666812ac98","read_only":false,"user_id":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","user_identity":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a 206a2db080e942a292df932654e528e5 - - -","instance":"[instance: 6bd76ba0-ac26-4954-9613-e39c3bb081be] ","version":"unknown","read_deleted":"no","logger":"nova.compute.manager","project_id":"206a2db080e942a292df932654e528e5","type":"nova","project_name":"gredadmin","tags":"nova-compute","timestamp":"2017-05-05T18:11:44.031234","remote_address":"10.39.75.15","quota_class":null,"host":"gredhos2-cp1-comp0002-mgmt","is_admin":true,"user":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","service_catalog":[{"endpoints":[{"adminURL":"https:\\\\/\\\\/10.39.75.51:8776\\\\/v1\\\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\\\/\\\\/10.39.75.51:8776\\\\/v1\\\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\\\/\\\\/gredcloud2.gene.com:8776\\\\/v1\\\\/206a2db080e942a292df932654e528e5"}],"type":"volume","name":"cinder"},{"endpoints":[{"adminURL":"https:\\\\/\\\\/10.39.75.51:8776\\\\/v2\\\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\\\/\\\\/10.39.75.51:8776\\\\/v2\\\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\\\/\\\\/gredcloud2.gene.com:8776\\\\/v2\\\\/206a2db080e942a292df932654e528e5"}],"type":"volumev2","name":"cinderv2"},{"endpoints":[{"adminURL":"https:\\\\/\\\\/10.39.75.51:9311","region":"region1","internalURL":"https:\\\\/\\\\/10.39.75.51:9311","publicURL":"https:\\\\/\\\\/gredcloud2.gene.com:9311"}],"type":"key-manager","name":"barbican"}],"path":"\\\\/opt\\\\/stack\\\\/venv\\\\/nova-20170214T094710Z\\\\/lib\\\\/python2.7\\\\/site-packages\\\\/nova\\\\/compute\\\\/manager.py","extra_keys":["instance","project","version"],"instance_lock_checked":false,"tenant":"206a2db080e942a292df932654e528e5","resource":"","show_deleted":false,"roles":["admin"],"user_name":"simran","project":"unknown","project_domain":null,"request_id":"req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06","user_domain":null,"@version":"1","levelname":"WARNING"},"dimensions":{"service":"nova","message_format_source":"json","message_format":"json","type":"nova-compute","@timestamp":"2017-05-05T18:20:47.495Z","control_plane":"control-plane-1","cluster":"compute","host":"gredhos2-cp1-comp0002-mgmt","file":"\\\\/var\\\\/log\\\\/nova\\\\/nova-compute-json.log","@version":"1","cloud_name":"gredcloud2"}}}')])] _send_broker_aware_request /opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207 /var/log/kronos/kronos-api.log:2017-05-05 18:20:47.799 17000 DEBUG kafka.client [-] Request 931074 to BrokerMetadata(nodeId=2, host='10.39.75.17', port=9092): [ProduceRequest(topic='logs', partition=120, messages=[Message(magic=0, attributes=0, key='1494008447799', value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008447,"log":{"message":{"domain":null,"color":"","@timestamp":"2017-05-05T18:20:47.717Z","isotime":"2017-05-05T18:20:47.717599+00:00","resource_uuid":null,"message":"Failed to revert task state for instance. Error: Timed out waiting for a reply to message ID 9ec8ad37f9504e549855a6331757da8a","read_only":false,"user_id":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","user_identity":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a 206a2db080e942a292df932654e528e5 - - -","instance":"[instance: 6bd76ba0-ac26-4954-9613-e39c3bb081be] ","version":"unknown","read_deleted":"no","logger":"nova.compute.manager","project_id":"206a2db080e942a292df932654e528e5","type":"nova","project_name":"gredadmin","tags":"nova-compute","timestamp":"2017-05-05T18:11:44.031234","remote_address":"10.39.75.15","quota_class":null,"host":"gredhos2-cp1-comp0006-mgmt","is_admin":true,"user":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","service_catalog":[{"endpoints":[{"adminURL":"https:\\/\\/10.39.75.51:8776\\/v1\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\/\\/10.39.75.51:8776\\/v1\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\/\\/gredcloud2.gene.com:8776\\/v1\\/206a2db080e942a292df932654e528e5"}],"type":"volume","name":"cinder"},{"endpoints":[{"adminURL":"https:\\/\\/10.39.75.51:8776\\/v2\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\/\\/10.39.75.51:8776\\/v2\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\/\\/gredcloud2.gene.com:8776\\/v2\\/206a2db080e942a292df932654e528e5"}],"type":"volumev2","name":"cinderv2"},{"endpoints":[{"adminURL":"https:\\/\\/10.39.75.51:9311","region":"region1","internalURL":"https:\\/\\/10.39.75.51:9311","publicURL":"https:\\/\\/gredcloud2.gene.com:9311"}],"type":"key-manager","name":"barbican"}],"path":"\\/opt\\/stack\\/venv\\/nova-20170214T094710Z\\/lib\\/python2.7\\/site-packages\\/nova\\/compute\\/manager.py","extra_keys":["instance","project","version"],"instance_lock_checked":false,"tenant":"206a2db080e942a292df932654e528e5","resource":"","show_deleted":false,"roles":["admin"],"user_name":"simran","project":"unknown","project_domain":null,"request_id":"req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06","user_domain":null,"@version":"1","levelname":"WARNING"},"dimensions":{"service":"nova","message_format_source":"json","message_format":"json","type":"nova-compute","@timestamp":"2017-05-05T18:20:47.724Z","control_plane":"control-plane-1","cluster":"compute","host":"gredhos2-cp1-comp0006-mgmt","file":"\\/var\\/log\\/nova\\/nova-compute-json.log","@version":"1","cloud_name":"gredcloud2"}}}')])] _send_broker_aware_request /opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207 /var/log/kronos/wsgi.log:[Fri May 05 18:20:47.800154 2017] [wsgi:error] [pid 17000:tid 140441909200640] 2017-05-05 18:20:47.799 17000 DEBUG kafka.client [-] Request 931074 to BrokerMetadata(nodeId=2, host='10.39.75.17', port=9092): [ProduceRequest(topic='logs', partition=120, messages=[Message(magic=0, attributes=0, key='1494008447799', value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008447,"log":{"message":{"domain":null,"color":"","@timestamp":"2017-05-05T18:20:47.717Z","isotime":"2017-05-05T18:20:47.717599+00:00","resource_uuid":null,"message":"Failed to revert task state for instance. Error: Timed out waiting for a reply to message ID 9ec8ad37f9504e549855a6331757da8a","read_only":false,"user_id":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","user_identity":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a 206a2db080e942a292df932654e528e5 - - -","instance":"[instance: 6bd76ba0-ac26-4954-9613-e39c3bb081be] ","version":"unknown","read_deleted":"no","logger":"nova.compute.manager","project_id":"206a2db080e942a292df932654e528e5","type":"nova","project_name":"gredadmin","tags":"nova-compute","timestamp":"2017-05-05T18:11:44.031234","remote_address":"10.39.75.15","quota_class":null,"host":"gredhos2-cp1-comp0006-mgmt","is_admin":true,"user":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","service_catalog":[{"endpoints":[{"adminURL":"https:\\\\/\\\\/10.39.75.51:8776\\\\/v1\\\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\\\/\\\\/10.39.75.51:8776\\\\/v1\\\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\\\/\\\\/gredcloud2.gene.com:8776\\\\/v1\\\\/206a2db080e942a292df932654e528e5"}],"type":"volume","name":"cinder"},{"endpoints":[{"adminURL":"https:\\\\/\\\\/10.39.75.51:8776\\\\/v2\\\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\\\/\\\\/10.39.75.51:8776\\\\/v2\\\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\\\/\\\\/gredcloud2.gene.com:8776\\\\/v2\\\\/206a2db080e942a292df932654e528e5"}],"type":"volumev2","name":"cinderv2"},{"endpoints":[{"adminURL":"https:\\\\/\\\\/10.39.75.51:9311","region":"region1","internalURL":"https:\\\\/\\\\/10.39.75.51:9311","publicURL":"https:\\\\/\\\\/gredcloud2.gene.com:9311"}],"type":"key-manager","name":"barbican"}],"path":"\\\\/opt\\\\/stack\\\\/venv\\\\/nova-20170214T094710Z\\\\/lib\\\\/python2.7\\\\/site-packages\\\\/nova\\\\/compute\\\\/manager.py","extra_keys":["instance","project","version"],"instance_lock_checked":false,"tenant":"206a2db080e942a292df932654e528e5","resource":"","show_deleted":false,"roles":["admin"],"user_name":"simran","project":"unknown","project_domain":null,"request_id":"req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06","user_domain":null,"@version":"1","levelname":"WARNING"},"dimensions":{"service":"nova","message_format_source":"json","message_format":"json","type":"nova-compute","@timestamp":"2017-05-05T18:20:47.724Z","control_plane":"control-plane-1","cluster":"compute","host":"gredhos2-cp1-comp0006-mgmt","file":"\\\\/var\\\\/log\\\\/nova\\\\/nova-compute-json.log","@version":"1","cloud_name":"gredcloud2"}}}')])] _send_broker_aware_request /opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207 /var/log/nova/nova-conductor.log:2017-05-05 18:18:47.419 7255 ERROR nova.scheduler.utils [req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06 07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a 206a2db080e942a292df932654e528e5] [instance: 6bd76ba0-ac26-4954-9613-e39c3bb081be] Error from last host: gredhos2-cp1-comp0002-mgmt (node gredhos2-cp1-comp0002-mgmt): [u'Traceback (most recent call last):\n', u' File "/opt/stack/venv/nova-20170214T094710Z/lib/python2.7/site-packages/nova/compute/manager.py", line 1930, in _do_build_and_run_instance\n filter_properties)\n', u' File "/opt/stack/venv/nova-20170214T094710Z/lib/python2.7/site-packages/nova/compute/manager.py", line 2120, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 6bd76ba0-ac26-4954-9613-e39c3bb081be was re-scheduled: Timed out waiting for a reply to message ID b2c5a1fbfe4e48c38954f6ba9bb6e04b\n'] /var/log/nova/nova-conductor-json.log:{"domain": null, "color": "", "@timestamp": "2017-05-05T18:18:47.419Z", "isotime": "2017-05-05T18:18:47.419068+00:00", "resource_uuid": null, "message": "Error from last host: gredhos2-cp1-comp0002-mgmt (node gredhos2-cp1-comp0002-mgmt): [u'Traceback (most recent call last):\\n', u' File \"/opt/stack/venv/nova-20170214T094710Z/lib/python2.7/site-packages/nova/compute/manager.py\", line 1930, in _do_build_and_run_instance\\n filter_properties)\\n', u' File \"/opt/stack/venv/nova-20170214T094710Z/lib/python2.7/site-packages/nova/compute/manager.py\", line 2120, in _build_and_run_instance\\n instance_uuid=instance.uuid, reason=six.text_type(e))\\n', u'RescheduledException: Build of instance 6bd76ba0-ac26-4954-9613-e39c3bb081be was re-scheduled: Timed out waiting for a reply to message ID b2c5a1fbfe4e48c38954f6ba9bb6e04b\\n']", "read_only": false, "user_id": "07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a", "user_identity": "07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a 206a2db080e942a292df932654e528e5 - - -", "instance": "[instance: 6bd76ba0-ac26-4954-9613-e39c3bb081be] ", "version": "unknown", "read_deleted": "no", "logger": "nova.scheduler.utils", "project_id": "206a2db080e942a292df932654e528e5", "type": "nova", "project_name": "gredadmin", "tags": "nova-conductor", "timestamp": "2017-05-05T18:11:44.031234", "remote_address": "10.39.75.15", "quota_class": null, "host": "gredhos2-cp1-c1-m3-mgmt", "is_admin": true, "user": "07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a", "service_catalog": [{"endpoints": [{"adminURL": "https://10.39.75.51:8776/v1/206a2db080e942a292df932654e528e5", "region": "region1", "internalURL": "https://10.39.75.51:8776/v1/206a2db080e942a292df932654e528e5", "publicURL": "https://gredcloud2.gene.com:8776/v1/206a2db080e942a292df932654e528e5"}], "type": "volume", "name": "cinder"}, {"endpoints": [{"adminURL": "https://10.39.75.51:8776/v2/206a2db080e942a292df932654e528e5", "region": "region1", "internalURL": "https://10.39.75.51:8776/v2/206a2db080e942a292df932654e528e5", "publicURL": "https://gredcloud2.gene.com:8776/v2/206a2db080e942a292df932654e528e5"}], "type": "volumev2", "name": "cinderv2"}, {"endpoints": [{"adminURL": "https://10.39.75.51:9311", "region": "region1", "internalURL": "https://10.39.75.51:9311", "publicURL": "https://gredcloud2.gene.com:9311"}], "type": "key-manager", "name": "barbican"}], "path": "/opt/stack/venv/nova-20170214T094710Z/lib/python2.7/site-packages/nova/scheduler/utils.py", "extra_keys": ["instance_uuid", "project", "version"], "instance_lock_checked": false, "tenant": "206a2db080e942a292df932654e528e5", "instance_uuid": "6bd76ba0-ac26-4954-9613-e39c3bb081be", "resource": "", "show_deleted": false, "roles": ["admin"], "user_name": "simran", "project": "unknown", "project_domain": null, "request_id": "req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06", "user_domain": null, "@version": "1", "levelname": "ERROR"} /var/log/nova/nova-compute.log:2017-05-05 18:20:47.425 1816 WARNING nova.compute.manager [req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06 07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a 206a2db080e942a292df932654e528e5] [instance: 6bd76ba0-ac26-4954-9613-e39c3bb081be] Failed to revert task state for instance. Error: Timed out waiting for a reply to message ID a76c427f58744af7bb512c666812ac98 /var/log/nova/nova-compute-json.log:{"domain": null, "color": "", "@timestamp": "2017-05-05T18:20:47.425Z", "isotime": "2017-05-05T18:20:47.425718+00:00", "resource_uuid": null, "message": "Failed to revert task state for instance. Error: Timed out waiting for a reply to message ID a76c427f58744af7bb512c666812ac98", "read_only": false, "user_id": "07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a", "user_identity": "07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a 206a2db080e942a292df932654e528e5 - - -", "instance": "[instance: 6bd76ba0-ac26-4954-9613-e39c3bb081be] ", "version": "unknown", "read_deleted": "no", "logger": "nova.compute.manager", "project_id": "206a2db080e942a292df932654e528e5", "type": "nova", "project_name": "gredadmin", "tags": "nova-compute", "timestamp": "2017-05-05T18:11:44.031234", "remote_address": "10.39.75.15", "quota_class": null, "host": "gredhos2-cp1-comp0002-mgmt", "is_admin": true, "user": "07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a", "service_catalog": [{"endpoints": [{"adminURL": "https://10.39.75.51:8776/v1/206a2db080e942a292df932654e528e5", "region": "region1", "internalURL": "https://10.39.75.51:8776/v1/206a2db080e942a292df932654e528e5", "publicURL": "https://gredcloud2.gene.com:8776/v1/206a2db080e942a292df932654e528e5"}], "type": "volume", "name": "cinder"}, {"endpoints": [{"adminURL": "https://10.39.75.51:8776/v2/206a2db080e942a292df932654e528e5", "region": "region1", "internalURL": "https://10.39.75.51:8776/v2/206a2db080e942a292df932654e528e5", "publicURL": "https://gredcloud2.gene.com:8776/v2/206a2db080e942a292df932654e528e5"}], "type": "volumev2", "name": "cinderv2"}, {"endpoints": [{"adminURL": "https://10.39.75.51:9311", "region": "region1", "internalURL": "https://10.39.75.51:9311", "publicURL": "https://gredcloud2.gene.com:9311"}], "type": "key-manager", "name": "barbican"}], "path": "/opt/stack/venv/nova-20170214T094710Z/lib/python2.7/site-packages/nova/compute/manager.py", "extra_keys": ["instance", "project", "version"], "instance_lock_checked": false, "tenant": "206a2db080e942a292df932654e528e5", "resource": "", "show_deleted": false, "roles": ["admin"], "user_name": "simran", "project": "unknown", "project_domain": null, "request_id": "req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06", "user_domain": null, "@version": "1", "levelname": "WARNING"} /var/log/nova/nova-compute-json.log:{"domain": null, "color": "", "@timestamp": "2017-05-05T18:20:47.717Z", "isotime": "2017-05-05T18:20:47.717599+00:00", "resource_uuid": null, "message": "Failed to revert task state for instance. Error: Timed out waiting for a reply to message ID 9ec8ad37f9504e549855a6331757da8a", "read_only": false, "user_id": "07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a", "user_identity": "07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a 206a2db080e942a292df932654e528e5 - - -", "instance": "[instance: 6bd76ba0-ac26-4954-9613-e39c3bb081be] ", "version": "unknown", "read_deleted": "no", "logger": "nova.compute.manager", "project_id": "206a2db080e942a292df932654e528e5", "type": "nova", "project_name": "gredadmin", "tags": "nova-compute", "timestamp": "2017-05-05T18:11:44.031234", "remote_address": "10.39.75.15", "quota_class": null, "host": "gredhos2-cp1-comp0006-mgmt", "is_admin": true, "user": "07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a", "service_catalog": [{"endpoints": [{"adminURL": "https://10.39.75.51:8776/v1/206a2db080e942a292df932654e528e5", "region": "region1", "internalURL": "https://10.39.75.51:8776/v1/206a2db080e942a292df932654e528e5", "publicURL": "https://gredcloud2.gene.com:8776/v1/206a2db080e942a292df932654e528e5"}], "type": "volume", "name": "cinder"}, {"endpoints": [{"adminURL": "https://10.39.75.51:8776/v2/206a2db080e942a292df932654e528e5", "region": "region1", "internalURL": "https://10.39.75.51:8776/v2/206a2db080e942a292df932654e528e5", "publicURL": "https://gredcloud2.gene.com:8776/v2/206a2db080e942a292df932654e528e5"}], "type": "volumev2", "name": "cinderv2"}, {"endpoints": [{"adminURL": "https://10.39.75.51:9311", "region": "region1", "internalURL": "https://10.39.75.51:9311", "publicURL": "https://gredcloud2.gene.com:9311"}], "type": "key-manager", "name": "barbican"}], "path": "/opt/stack/venv/nova-20170214T094710Z/lib/python2.7/site-packages/nova/compute/manager.py", "extra_keys": ["instance", "project", "version"], "instance_lock_checked": false, "tenant": "206a2db080e942a292df932654e528e5", "resource": "", "show_deleted": false, "roles": ["admin"], "user_name": "simran", "project": "unknown", "project_domain": null, "request_id": "req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06", "user_domain": null, "@version": "1", "levelname": "WARNING"} /var/log/nova/nova-compute.log:2017-05-05 18:20:47.717 1943 WARNING nova.compute.manager [req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06 07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a 206a2db080e942a292df932654e528e5] [instance: 6bd76ba0-ac26-4954-9613-e39c3bb081be] Failed to revert task state for instance. Error: Timed out waiting for a reply to message ID 9ec8ad37f9504e549855a6331757da8a -- Thanks in advance! _______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack