Can you provide some more information on your environment, such as: 1) Version of Ambari 2) Whether the environment is kerberized 3) Are you running the Ambari agent as root, or another user. 4) Any information from the ambari-agent.log file that might seen to indicate a problem 5) You said that restarting the agents resolves the issue. Does it continue to happen after restarting? If so, how long before new warnings start to show up.
I’m guessing you have a kerberized environment running Ambari 2.0. Ambari will use curl in this case to attempt to make a connection to the web endpoints. It uses the keytabs and principals defined on the alert definition. For NameNode, as an example, it would use: hdfs-site/dfs.web.authentication.kerberos.keytab hdfs-site/dfs.web.authentication.kerberos.principal You’ll want to verify that these properties are correctly set and that the keytab file is accessible to the agent user. It could also be a cache problem as the agents cache the kerberos credentials in the agent’s temp directory. How long it takes after alerts start producing warnings would in determining if it’s a caching issue. On May 5, 2015, at 10:49 AM, Chanel Loïc <loic.cha...@worldline.com<mailto:loic.cha...@worldline.com>> wrote: Hi, I have currently some issues with cluster nodes. According to the Ambari web User Interface, I have 11 alerts linked to the fact that all nodes (except the monitoring one) return HTTP 403 response for services such as NameNode web UI, DataNode web UI, or NodeManager Health. The first weird thing is the fact that I can easily see that the so-called Forbidden ports are actually quite available (via cURL for example) and indicate that the cluster is totally ok. The second is the fact that the 11 alerts magically disappear when rebooting the agent on the hosts related to the errors. Does anyone know where these errors might come from ? Thanks, Loïc ________________________________ Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de ses destinataires. Il peut également être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire. L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité de Worldline ne pourra être recherchée quant au contenu de ce message. Bien que les meilleurs efforts soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne saurait être recherchée pour tout dommage résultant d'un virus transmis. This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Worldline liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.
