Hi all, I have a tribe node setup across two clusters, on server0 ($IP0) and server1 ($IP1), using unicast discovery. server0 also runs the elasticsearch tribe node instance.
This setup works fine most of the time, and I can see logs from all clusters on tribe node Kibana. The issue is that sometimes the elasticsearch instances go unresponsive. More specifically, it's the nodes API (including /_cat/nodes) that do not respond, which results in Kibana throwing the following error message: Could not reach http://.../_nodes. If you are using a proxy, ensure it is > configured correctly At first, I thought it was just the tribe node that was doing this, but just today I noticed this on one of the clusters as well. Other URLs don't seem to be affected (e.g. /, /_cluster/health, etc). After some time, the nodes API comes back and produces the following output, and goes unresponsive again. I've also tried restarting the elasticsearch processes. While it fixes the unresponsiveness in the short term, it then goes back to the same cycle. ======== jun@server0 ~ % curl -XGET "$IP0:9201/_cat/nodes?pretty=true&v=true" host ip heap.percent ram.percent load node.role master name server0 $IP0 11 5 2.97 c x tribe server1 $IP1 16 38 2.74 d x s1-elasticsearch server0 $IP0 17 5 2.97 d x s0-elasticsearch server1 $IP1 c x s1-logstash server0 $IP0 c x tribe server0 $IP0 c x tribe server0 $IP0 c x tribe server0 $IP0 c x s0-logstash jun@server0 ~ % curl -XGET "$IP0:9200/_cat/nodes?pretty=true&v=true" { "error" : "NullPointerException[null]", "status" : 500 } jun@server0 ~ % curl -XGET "$IP1:9200/_cat/nodes?pretty=true&v=true" { "error" : "NullPointerException[null]", "status" : 500 } jun@server0 ~ % curl -XGET '$IP0:9201/_nodes/process?pretty=true&v=true' { "cluster_name" : "elasticsearch", "nodes" : { "gPX5VOT5TdGPOF6wq9ErKA" : { "name" : "tribe", "transport_address" : "inet[/$IP0:9300]", "host" : "server0", "ip" : "$IP0", "version" : "1.3.4", "build" : "a70f3cc", "http_address" : "inet[/$IP0:9201]", "attributes" : { "client" : "true", "data" : "false" }, "process" : { "refresh_interval_in_millis" : 1000, "id" : 187050, "max_file_descriptors" : 65535, "mlockall" : false } }, "liMvaS8cSu2PafCM653LHA" : { "name" : "s1-elasticsearch", "transport_address" : "inet[/$IP1:9301]", "host" : "server1", "ip" : "$IP1", "version" : "1.3.4", "build" : "a70f3cc", "http_address" : "inet[/$IP1:9200]", "process" : { "refresh_interval_in_millis" : 1000, "id" : 32465, "max_file_descriptors" : 65535, "mlockall" : false } }, "z-GOP2v7Q7-FvlaS8bIBJQ" : { "name" : "s0-elasticsearch", "transport_address" : "inet[/$IP0:9301]", "host" : "server0", "ip" : "$IP0", "version" : "1.3.4", "build" : "a70f3cc", "http_address" : "inet[/$IP0:9200]", "process" : { "refresh_interval_in_millis" : 1000, "id" : 165512, "max_file_descriptors" : 65535, "mlockall" : false } } } } ======== Any help would be greatly appreciated. Thanks, Jun -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/314fd788-a883-45c6-9c94-226dd29f9d5c%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
