OK, so I think I have a clear picture of how you get to this situation. I’d still like to know a few things:
1) When you have the warnings present in the web client, can you try the alerts URL I posted below to see the actual alerts coming back from the API. I’m mostly interesting in whether any come back in the first place, and what the most recent timestamp was (indicating they are actually running and still reporting a warning status) 2) When the alerts are present in the web client, do you see any output in the agent log file that I mentioned for alerts that start with [Alert]. On May 6, 2015, at 5:14 AM, Chanel Loïc <loic.cha...@worldline.com<mailto:loic.cha...@worldline.com>> wrote: I encounter the warnings and critical alerts when deploying the cluster. I install Ambari server 2.0 on a VM and Ambari agent 2.0 on 4 others VMs. Then, I give the Ambari server a blueprint coming from a functioning cluster to instantiate my new cluster and have a quickstart configuration. Then, when logging into the Ambari web application to ensure everything is running properly, I have these alerts concerning the HTTP 403 errors returned by all host VMs but the one which only handles Ambari metrics. As I am not sure my explanations are quite understandable, do not hesitate to tell me if something remains unclear. Thanks, De : Jonathan Hurley [mailto:jhur...@hortonworks.com] Envoyé : mardi 5 mai 2015 20:38 À : user@ambari.apache.org<mailto:user@ambari.apache.org> Objet : Re: Restarting nodes to avoid HTTP 403 If restarting the agents fixes everything, can you explain when you first encounter the warnings? Is this only after a cluster deployment? You can also check to see if this is some sort of web client issue by issuing the following GET: http://server/api/v1/clusters/<cluster>/alerts?fields=*&Alert/state.in(CRITICAL,WARNING)<http://server/api/v1/clusters/%3Ccluster%3E/alerts?fields=*&Alert/state.in(CRITICAL,WARNING)> This will show you alerts which are actually being returned from the agents in a warning or critical state. For reference, you can also look in /var/log/ambari-agent/ambari-agent.log to see if you see any alert issues. Most important messages are prefixed with "[Alert]”. On May 5, 2015, at 11:43 AM, Chanel Loïc <loic.cha...@worldline.com<mailto:loic.cha...@worldline.com>> wrote: Indeed, I did not gave so much information about my problem, sorry about that. Here is your answers : 1) Version of Ambari --> I’m using Ambari 2.0 2) Whether the environment is kerberized --> Not it’s not. Kerberos security is not enabled on this cluster. 3) Are you running the Ambari agent as root, or another user --> I am running it as root 4) Any information from the ambari-agent.log file that might seem to indicate a problem --> Here is another weird thing I did not mentioned : I could not find logs referring to the problem. 5) You said that restarting the agents resolves the issue. Does it continue to happen after restarting? If so, how long before new warnings start to show up --> Restarting the agent totally resolves the problem. It does not happen anymore, and everything run quite normally. De : Jonathan Hurley [mailto:jhur...@hortonworks.com] Envoyé : mardi 5 mai 2015 17:36 À : user@ambari.apache.org<mailto:user@ambari.apache.org> Objet : Re: Restarting nodes to avoid HTTP 403 Can you provide some more information on your environment, such as: 1) Version of Ambari 2) Whether the environment is kerberized 3) Are you running the Ambari agent as root, or another user. 4) Any information from the ambari-agent.log file that might seen to indicate a problem 5) You said that restarting the agents resolves the issue. Does it continue to happen after restarting? If so, how long before new warnings start to show up. I’m guessing you have a kerberized environment running Ambari 2.0. Ambari will use curl in this case to attempt to make a connection to the web endpoints. It uses the keytabs and principals defined on the alert definition. For NameNode, as an example, it would use: hdfs-site/dfs.web.authentication.kerberos.keytab hdfs-site/dfs.web.authentication.kerberos.principal You’ll want to verify that these properties are correctly set and that the keytab file is accessible to the agent user. It could also be a cache problem as the agents cache the kerberos credentials in the agent’s temp directory. How long it takes after alerts start producing warnings would in determining if it’s a caching issue. On May 5, 2015, at 10:49 AM, Chanel Loïc <loic.cha...@worldline.com<mailto:loic.cha...@worldline.com>> wrote: Hi, I have currently some issues with cluster nodes. According to the Ambari web User Interface, I have 11 alerts linked to the fact that all nodes (except the monitoring one) return HTTP 403 response for services such as NameNode web UI, DataNode web UI, or NodeManager Health. The first weird thing is the fact that I can easily see that the so-called Forbidden ports are actually quite available (via cURL for example) and indicate that the cluster is totally ok. The second is the fact that the 11 alerts magically disappear when rebooting the agent on the hosts related to the errors. Does anyone know where these errors might come from ? Thanks, Loïc ________________________________ Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de ses destinataires. Il peut également être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire. L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité de Worldline ne pourra être recherchée quant au contenu de ce message. Bien que les meilleurs efforts soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne saurait être recherchée pour tout dommage résultant d'un virus transmis. This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Worldline liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. ________________________________ Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de ses destinataires. Il peut également être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire. L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité de Worldline ne pourra être recherchée quant au contenu de ce message. Bien que les meilleurs efforts soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne saurait être recherchée pour tout dommage résultant d'un virus transmis. This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Worldline liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. ________________________________ Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de ses destinataires. Il peut également être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire. L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité de Worldline ne pourra être recherchée quant au contenu de ce message. Bien que les meilleurs efforts soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne saurait être recherchée pour tout dommage résultant d'un virus transmis. This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Worldline liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.
