Srikanth Janardhan created AMBARI-24179: -------------------------------------------
Summary: Ambari Metrics Service check fails after deleting a host Key: AMBARI-24179 URL: https://issues.apache.org/jira/browse/AMBARI-24179 Project: Ambari Issue Type: Bug Components: ambari-metrics Affects Versions: 2.7.0 Reporter: Srikanth Janardhan Assignee: Dmytro Sen Fix For: 2.7.0 ambari metrics service check failed immediately after deleting a host: {code:java} stderr: Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/AMBARI_METRICS/package/scripts/service_check.py", line 304, in AMSServiceCheck().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 353, in execute method(env) File "/usr/lib/ambari-agent/lib/ambari_commons/os_family_impl.py", line 89, in thunk return fn(*args, **kwargs) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/AMBARI_METRICS/package/scripts/service_check.py", line 184, in service_check raise Fail("All metrics collectors are unavailable.") resource_management.core.exceptions.Fail: All metrics collectors are unavailable. stdout: 2018-06-25 04:42:25,088 - Using hadoop conf dir: /usr/hdp/3.0.0.0-1541/hadoop/conf 2018-06-25 04:42:25,095 - checked_call['hostid'] {} 2018-06-25 04:42:25,100 - checked_call returned (0, '1bac1213') 2018-06-25 04:42:25,102 - Ambari Metrics service check was started. 2018-06-25 04:42:25,121 - Generated metrics for host ctr-e138-1518143905142-378410-01-000009.hwx.site : { "metrics": [ { "metricname": "AMBARI_METRICS.SmokeTest.FakeMetric", "appid": "amssmoketestfake", "hostname": "ctr-e138-1518143905142-378410-01-000012.hwx.site", "starttime": 1529901745000, "metrics": { "1529901745000": 0.602995821312, "1529901746000": 1529901745000 } } ] } 2018-06-25 04:42:25,122 - Connecting (POST) to ctr-e138-1518143905142-378410-01-000009.hwx.site:6188/ws/v1/timeline/metrics/ 2018-06-25 04:42:25,132 - Http response for host ctr-e138-1518143905142-378410-01-000009.hwx.site: 200 OK 2018-06-25 04:42:25,133 - Http data: 2018-06-25 04:42:25,133 - Metrics were saved. 2018-06-25 04:42:25,133 - Connecting (GET) to ctr-e138-1518143905142-378410-01-000009.hwx.site:6188/ws/v1/timeline/metrics?metricNames=AMBARI_METRICS.SmokeTest.FakeMetric&hostname=ctr-e138-1518143905142-378410-01-000012.hwx.site&precision=seconds&grouped=false&startTime=1529901685000&appId=amssmoketestfake&endTime=1529901806000 2018-06-25 04:42:25,138 - Http response for host ctr-e138-1518143905142-378410-01-000009.hwx.site : 200 OK 2018-06-25 04:42:25,138 - Http data: {"metrics":[]} 2018-06-25 04:42:25,138 - Metrics were retrieved from host ctr-e138-1518143905142-378410-01-000009.hwx.site 2018-06-25 04:42:25,139 - Values weren't stored yet. Retrying in 10 seconds. 2018-06-25 04:42:35,154 - Http response for host ctr-e138-1518143905142-378410-01-000009.hwx.site : 200 OK 2018-06-25 04:42:35,154 - Http data: {"metrics":[]} 2018-06-25 04:42:35,154 - Metrics were retrieved from host ctr-e138-1518143905142-378410-01-000009.hwx.site 2018-06-25 04:42:35,155 - Values weren't stored yet. Retrying in 10 seconds. 2018-06-25 04:42:45,170 - Http response for host ctr-e138-1518143905142-378410-01-000009.hwx.site : 200 OK 2018-06-25 04:42:45,170 - Http data: {"metrics":[]} 2018-06-25 04:42:45,171 - Metrics were retrieved from host ctr-e138-1518143905142-378410-01-000009.hwx.site 2018-06-25 04:42:45,171 - Values weren't stored yet. Retrying in 10 seconds. 2018-06-25 04:42:55,186 - Http response for host ctr-e138-1518143905142-378410-01-000009.hwx.site : 200 OK 2018-06-25 04:42:55,186 - Http data: {"metrics":[]} 2018-06-25 04:42:55,187 - Metrics were retrieved from host ctr-e138-1518143905142-378410-01-000009.hwx.site 2018-06-25 04:42:55,187 - Values weren't stored yet. Retrying in 10 seconds. 2018-06-25 04:43:05,204 - Http response for host ctr-e138-1518143905142-378410-01-000009.hwx.site : 200 OK 2018-06-25 04:43:05,204 - Http data: {"metrics":[]} 2018-06-25 04:43:05,205 - Metrics were retrieved from host ctr-e138-1518143905142-378410-01-000009.hwx.site 2018-06-25 04:43:05,205 - Ambari Metrics service check failed on collector host ctr-e138-1518143905142-378410-01-000009.hwx.site. Reason : Values 0.602995821312 and 1529901745000 were not found in the response. 2018-06-25 04:43:05,207 - Exception while running function '>' for 'ctr-e138-1518143905142-378410-01-000009.hwx.site'. Reason : Ambari Metrics service check failed on collector host ctr-e138-1518143905142-378410-01-000009.hwx.site. Reason : Values 0.602995821312 and 1529901745000 were not found in the response. Command failed after 1 tries {code} *Subsequent service check passed, though.* The issue looks not related to the host deletion, seems like sometimes the data posted during service check is not saved by collector. -- This message was sent by Atlassian JIRA (v7.6.3#76005)