Hi, We are facing an intermittent issue while using the universal recommender template on our production servers. Every once in a while we get an error that none of the ElasticSearch nodes are reachable (org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: [])
Our elasticsearch cluster is hosted on a elasticsearch hosting provider(Qbox) and at the time of receiving this error all the shards were green and reachable from the server where predictio-io template is hosted. There was no shortage of RAM or disk space on that server when this issue occurred. The issue gets resolved on performing the following steps: 1) pio undeploy 2) pio-stop-all 3) pio-start-all 4) pio deploy Pio version used is 0.10.0-incubating and universal recommender version is 0.5.0 We did face this issue earlier as well and could not figure out why the ES nodes became unreachable after a particular period. The pio-env config for ES is below: PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=<cluster-name> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=<hosted-es-domain> PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=<port> // Transport Layer Port Used Here PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=<local-es-path> We are currently using ElasticSearch version 1.5.2 on our hosted servers as version 1.4.0 used by the pio is not supported on the hosted service. We are using the Transport Layer port to communicate with the ES nodes as opposed to calling the https endpoint. Please help us figure out the underlying cause for this issue. Thanks, Ashutosh
