Just tried 0.90.11. Same issue. I have two running nodes on ec2..and wow...they cant talk.
On Tue, Feb 4, 2014 at 6:51 AM, David Montgomery <davidmontgom...@gmail.com>wrote: > Hi Jonathan > > Just be clear...you could not cluster? I wili try the lastest stable > release cuase I am giving up on 1.0.0RC. > > Thanks > > > On Sun, Feb 2, 2014 at 10:10 AM, Jonathan Goodwin < > jonathanwgood...@gmail.com> wrote: > >> Hello David and David, >> >> I took a look at the logs / configs posted and mine are mostly similar >> (though i'm not using any rivers) and I'm having the same issue. >> I've tested on 0.90.9 and 0.90.7, I hadn't tried the 1.0.0RC yet because >> the logstash website suggested I stick to 0.90.9 >> If you need any more logs / config samples I'd be happy to provide those. >> >> Best, >> JG >> >> On Friday, January 31, 2014 3:46:07 PM UTC-5, David Montgomery wrote: >> >>> Hi, >>> >>> In chef I used -N flag which changes the host name when using >>> chef-client. I removed the flag and built again. No difference. I have >>> two nodes running. Both have the name cluster name, ports are 100% wide >>> open...and wow..ES cant talk despite both running in the same region. >>> >>> So...ES...any suggestions? >>> >>> root@ip-10-238-131-236:/var/log/elasticsearch# hostname >>> ip-10-238-131-236 >>> >>> >>> root@ip-10-238-131-236:/var/log/elasticsearch# curl -XGET ' >>> http://127.0.0.1:9200/_cluster/health?pretty=true' >>> { >>> "cluster_name" : "mizilaelasticsearch", >>> "status" : "green", >>> "timed_out" : false, >>> "number_of_nodes" : 1, >>> "number_of_data_nodes" : 1, >>> "active_primary_shards" : 0, >>> "active_shards" : 0, >>> "relocating_shards" : 0, >>> "initializing_shards" : 0, >>> "unassigned_shards" : 0 >>> } >>> >>> >>> >>> >>> >>> >>> On Fri, Jan 31, 2014 at 12:20 AM, Ivan Brusic <iv...@brusic.com> wrote: >>> >>>> Did you override your network.host configuration parameter by any >>>> chance? >>>> >>>> -- >>>> Ivan >>>> >>>> >>>> On Wed, Jan 29, 2014 at 1:56 PM, David Montgomery <davidmo...@gmail.com >>>> > wrote: >>>> >>>>> Hi, >>>>> >>>>> Details >>>>> >>>>> 1) both servers are sing the same sec group >>>>> 2 all ports between 9200 and 9300 are open 0.0.0.0/0 >>>>> 3) ec2 instacances in us-east region >>>>> 4) Did a fresh boot on 2 servers both servers are running. Both are >>>>> not talking >>>>> 5) Attached are the logs for both servers and the common >>>>> elasticsearch.yml file >>>>> 6) Below is he chef recipe I wrote. I am using 1.0.0.RC1 with >>>>> elasticsearch-cloud-aws/2.0.0.RC1 >>>>> /usr/share/elasticsearch/plugins/cloud-aws >>>>> ubuntu@ip-10-73-142-187:/usr/share/elasticsearch/plugins/cloud-aws$ ls >>>>> aws-java-sdk-1.3.32.jar commons-codec-1.4.jar >>>>> commons-logging-1.1.1.jar elasticsearch-cloud-aws-2.0.0.RC1.jar >>>>> httpclient-4.1.1.jar httpcore-4.1.jar >>>>> >>>>> >>>>> data_bag("my_data_bag") >>>>> db = data_bag_item("my_data_bag", "my") >>>>> AWS_ACCESS_KEY_ID = db[node.chef_environment][' >>>>> aws']['AWS_ACCESS_KEY_ID'] >>>>> AWS_SECRET_ACCESS_KEY = db[node.chef_environment][' >>>>> aws']['AWS_SECRET_ACCESS_KEY'] >>>>> >>>>> >>>>> version = '1.0.0.RC1' >>>>> remote_file >>>>> "#{Chef::Config[:file_cache_path]}/elasticsearch-#{version}.deb" >>>>> do >>>>> source "https://download.elasticsearch.org/ >>>>> elasticsearch/elasticsearch/elasticsearch-#{version}.deb" >>>>> action :create_if_missing >>>>> end >>>>> >>>>> dpkg_package >>>>> "#{Chef::Config[:file_cache_path]}/elasticsearch-#{version}.deb" >>>>> do >>>>> action :install >>>>> end >>>>> bash "add_aws" do >>>>> cwd "/usr/share/elasticsearch" >>>>> code <<-EOH >>>>> git clone https://github.com/elasticsearch/elasticsearch- >>>>> cloud-aws.git >>>>> /usr/share/elasticsearch/bin/plugin -install >>>>> elasticsearch/elasticsearch-cloud-aws/2.0.0.RC1 >>>>> EOH >>>>> not_if {File.exists?("/usr/share/elasticsearch/elasticsearch- >>>>> cloud-aws")} >>>>> end >>>>> >>>>> service "elasticsearch" do >>>>> supports :restart => true, :start => true, :stop => true >>>>> action [ :enable, :start] >>>>> end >>>>> >>>>> template "/etc/elasticsearch/logging.yml" do >>>>> path "/etc/elasticsearch/logging.yml" >>>>> source "logging.yml.erb" >>>>> owner "root" >>>>> group "root" >>>>> mode "0755" >>>>> notifies :restart, resources(:service => "elasticsearch") >>>>> end >>>>> >>>>> template "/etc/elasticsearch/elasticsearch.yml" do >>>>> path "/etc/elasticsearch/elasticsearch.yml" >>>>> source "elasticsearch.yml.erb" >>>>> owner "root" >>>>> group "root" >>>>> mode "0755" >>>>> variables :AWS_ACCESS_KEY_ID => "#{AWS_ACCESS_KEY_ID}", >>>>> :AWS_SECRET_ACCESS_KEY => "#{AWS_SECRET_ACCESS_KEY}", :region => >>>>> "us-east-1" >>>>> notifies :restart, resources(:service => "elasticsearch") >>>>> end >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Tue, Jan 28, 2014 at 6:40 AM, David Montgomery < >>>>> davidmo...@gmail.com> wrote: >>>>> >>>>>> sure..let me the logs together and will post two files ..one log file >>>>>> for each of the servers. >>>>>> >>>>>> Thanks >>>>>> >>>>>> >>>>>> On Tue, Jan 28, 2014 at 6:25 AM, David Pilato <da...@pilato.fr>wrote: >>>>>> >>>>>>> Is it a known problem? Not yet. That's the reason I'm asking for >>>>>>> logs. I would like to understand what is happening here. So full logs on >>>>>>> both nodes would help. >>>>>>> >>>>>>> The way you set unicast is fine to me. But please, could you share >>>>>>> with me (even through Direct mail) your elasticsearch.yml file and your >>>>>>> logs in TRACE level? >>>>>>> You can replace your credentials settings with XXXXXXXX. Please >>>>>>> don't change anything else. I'd like to reproduce your case. >>>>>>> >>>>>>> Thanks! >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> David ;-) >>>>>>> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs >>>>>>> >>>>>>> >>>>>>> Le 27 janv. 2014 à 22:57, David Montgomery <davidmo...@gmail.com> a >>>>>>> écrit : >>>>>>> >>>>>>> I have to say..it this a know problem? >>>>>>> >>>>>>> I removed the keys. >>>>>>> >>>>>>> So.. should the servers still cluster? Well....its >>>>>>> not.........again given that the two machine can talk to each other via >>>>>>> telent, in the same region, same sec group restarted services etc....no >>>>>>> go......no cluster >>>>>>> >>>>>>> >>>>>>> So....how do I manually configure nodes? I have given up on auto >>>>>>> discover. Clearly does now work for me. >>>>>>> >>>>>>> Is the below an example? >>>>>>> >>>>>>> discovery.zen.ping.multicast.enabled: false >>>>>>> discovery.zen.ping.unicast.hosts: ["0.test.com:9200", "1.test.com >>>>>>> :9200"] >>>>>>> >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Tuesday, January 28, 2014 4:19:25 AM UTC+8, David Pilato wrote: >>>>>>>> >>>>>>>> Yes. Could you gist it (and remove first keys / secrets of course)? >>>>>>>> >>>>>>>> -- >>>>>>>> David ;-) >>>>>>>> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs >>>>>>>> >>>>>>>> >>>>>>>> Le 27 janv. 2014 à 20:53, David Montgomery <davidmo...@gmail.com> >>>>>>>> a écrit : >>>>>>>> >>>>>>>> I assume that means this in the logging.yml fike? >>>>>>>> # discovery >>>>>>>> discovery: TRACE >>>>>>>> >>>>>>>> I do have it on...what should I be looking for in the logs? >>>>>>>> >>>>>>>> On Tuesday, January 28, 2014 3:47:10 AM UTC+8, David Montgomery >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> As an fyi..I can telnet into the other machine. Its only ES that >>>>>>>>> does not work despite following the online docs. >>>>>>>>> >>>>>>>>> If ES aws auto discovery does not work l assuming assume that >>>>>>>>> http://www.elasticsearch.org/guide/en/elasticsearch/referenc >>>>>>>>> e/current/modules-discovery-zen.html will work? >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tuesday, January 28, 2014 1:46:20 AM UTC+8, David Pilato wrote: >>>>>>>>>> >>>>>>>>>> Hi David, >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I think you should have nodes running under the same security >>>>>>>>>> group. In that case, you don't have to open 9300-9400 ports to >>>>>>>>>> public. >>>>>>>>>> If they don't belong to the same group or if you want to reach >>>>>>>>>> them from outside ec2 platform (public IP), then you need to set >>>>>>>>>> host_type >>>>>>>>>> to public_ip. >>>>>>>>>> >>>>>>>>>> It sounds like your nodes are trying to connect with private ip >>>>>>>>>> here (which is default). >>>>>>>>>> >>>>>>>>>> My 2 cents >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> *David Pilato* | *Technical Advocate* | *Elasticsearch.com >>>>>>>>>> <http://Elasticsearch.com>* >>>>>>>>>> @dadoonet <https://twitter.com/dadoonet> | >>>>>>>>>> @elasticsearchfr<https://twitter.com/elasticsearchfr> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Le 27 janvier 2014 at 18:07:48, David Montgomery ( >>>>>>>>>> davidmo...@gmail.com) a écrit: >>>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I have two nodes runnging on ec2. >>>>>>>>>> >>>>>>>>>> curl -X GET http://localhost:9200/_cluster/health?pretty=true >>>>>>>>>> { >>>>>>>>>> "cluster_name" : "12345elasticsearch", >>>>>>>>>> "status" : "green", >>>>>>>>>> "timed_out" : false, >>>>>>>>>> "number_of_nodes" : 1, >>>>>>>>>> "number_of_data_nodes" : 1, >>>>>>>>>> "active_primary_shards" : 0, >>>>>>>>>> "active_shards" : 0, >>>>>>>>>> "relocating_shards" : 0, >>>>>>>>>> "initializing_shards" : 0, >>>>>>>>>> "unassigned_shards" : 0 >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> So..its fair to say that ec2 clustering did not work given the >>>>>>>>>> below config. Yes...the aws keys are correct and I have port >>>>>>>>>> 9200-9400 >>>>>>>>>> open. What elasticsearch advertiseres is that all that is needed. >>>>>>>>>> >>>>>>>>>> I am using 1.0.0.RC1 with elasticsearch-cloud-aws= 2.0.0.RC1 on >>>>>>>>>> ubuntu 12.04 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> cloud.aws.access_key: <%=@AWS_ACCESS_KEY_ID%> >>>>>>>>>> cloud.aws.secret_key: <%=@AWS_SECRET_ACCESS_KEY%> >>>>>>>>>> discovery.type: ec2 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> In the logs I see the below: >>>>>>>>>> >>>>>>>>>> [2014-01-27 16:58:49,708][TRACE][discovery.zen.ping.unicast] >>>>>>>>>> [Administrator] [1] failed to connect to [#cloud-i-1079ac6b-0][aws- >>>>>>>>>> elasticsearch-east-development-20140127160434][inet[/10.211. >>>>>>>>>> 29.210:9300]] >>>>>>>>>> org.elasticsearch.transport.ConnectTransportException: [][inet[/ >>>>>>>>>> 10.211.29.210:9300]] connect_timeout[30s] >>>>>>>>>> at org.elasticsearch.transport.netty.NettyTransport. >>>>>>>>>> connectToChannelsLight(NettyTransport.java:676) >>>>>>>>>> at org.elasticsearch.transport.netty.NettyTransport. >>>>>>>>>> connectToNode(NettyTransport.java:636) >>>>>>>>>> at org.elasticsearch.transport.netty.NettyTransport. >>>>>>>>>> connectToNodeLight(NettyTransport.java:603) >>>>>>>>>> at org.elasticsearch.transport.TransportService. >>>>>>>>>> connectToNodeLight(TransportService.java:133) >>>>>>>>>> at org.elasticsearch.discovery.zen.ping.unicast. >>>>>>>>>> UnicastZenPing$3.run(UnicastZenPing.java:278) >>>>>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker( >>>>>>>>>> ThreadPoolExecutor.java:1146) >>>>>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>>>>>>>> ThreadPoolExecutor.java:615) >>>>>>>>>> at java.lang.Thread.run(Thread.java:701) >>>>>>>>>> Caused by: org.elasticsearch.common.netty.channel. >>>>>>>>>> ConnectTimeoutException: connection timed out: /10.239.36.98:9300 >>>>>>>>>> at org.elasticsearch.common.netty.channel.socket.nio. >>>>>>>>>> NioClientBoss.processConnectTimeout(NioClientBoss.java:137) >>>>>>>>>> at org.elasticsearch.common.netty.channel.socket.nio. >>>>>>>>>> NioClientBoss.process(NioClientBoss.java:83) >>>>>>>>>> at org.elasticsearch.common.netty.channel.socket.nio. >>>>>>>>>> AbstractNioSelector.run(AbstractNioSelector.java:318) >>>>>>>>>> at org.elasticsearch.common.netty.channel.socket.nio. >>>>>>>>>> NioClientBoss.run(NioClientBoss.java:42) >>>>>>>>>> at org.elasticsearch.common.netty.util. >>>>>>>>>> ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) >>>>>>>>>> at org.elasticsearch.common.netty.util.internal. >>>>>>>>>> DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) >>>>>>>>>> ... 3 more >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>> Google Groups "elasticsearch" group. >>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to elasticsearc...@googlegroups.com. >>>>>>>>>> To view this discussion on the web visit >>>>>>>>>> https://groups.google.com/d/msgid/elasticsearch/ec028f53-c6b >>>>>>>>>> 2-48e1-b714-a6f1ac7ae1cd%40googlegroups.com. >>>>>>>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>>>>>>> >>>>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "elasticsearch" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to elasticsearc...@googlegroups.com. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/elasticsearch/4d1c068a-3e9 >>>>>>>> c-4731-84e2-62eeeb63fa81%40googlegroups.com. >>>>>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "elasticsearch" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to elasticsearc...@googlegroups.com. >>>>>>> >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/elasticsearch/2bdefbe0- >>>>>>> 3ab4-4b85-b473-8b502b812d5b%40googlegroups.com. >>>>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to a topic in >>>>>>> the Google Groups "elasticsearch" group. >>>>>>> To unsubscribe from this topic, visit https://groups.google.com/d/ >>>>>>> topic/elasticsearch/PDDOu42M9Mc/unsubscribe. >>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>> elasticsearc...@googlegroups.com. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/elasticsearch/2804A7F6- >>>>>>> B2B8-4C23-B273-B7B7E0F805E0%40pilato.fr. >>>>>>> >>>>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>>>> >>>>>> >>>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "elasticsearch" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to elasticsearc...@googlegroups.com. >>>>> To view this discussion on the web visit https://groups.google.com/d/ >>>>> msgid/elasticsearch/CACF%2B8roDUwpq--dH9eAUoo-Xe-GEtgH% >>>>> 2B7WDJrbgjPWJBV2ijFQ%40mail.gmail.com. >>>>> >>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>> >>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "elasticsearch" group. >>>> To unsubscribe from this topic, visit https://groups.google.com/d/ >>>> topic/elasticsearch/PDDOu42M9Mc/unsubscribe. >>>> To unsubscribe from this group and all its topics, send an email to >>>> elasticsearc...@googlegroups.com. >>>> To view this discussion on the web visit https://groups.google.com/d/ >>>> msgid/elasticsearch/CALY%3DcQDHsDjF0AGUAoC3rLz_ >>>> y5eGvFPtDEooRj0xP87mRjxurg%40mail.gmail.com. >>>> >>>> For more options, visit https://groups.google.com/groups/opt_out. >>>> >>> >>> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "elasticsearch" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/elasticsearch/PDDOu42M9Mc/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> elasticsearch+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/a48fe555-0b95-4281-ba9e-351b21f4a9e9%40googlegroups.com >> . >> >> For more options, visit https://groups.google.com/groups/opt_out. >> > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CACF%2B8ro6cwQ1pw6nbMo3ciX85q7Gen5s%2BS%2B1THuebiM130dihQ%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
