Hello David and David, I took a look at the logs / configs posted and mine are mostly similar (though i'm not using any rivers) and I'm having the same issue. I've tested on 0.90.9 and 0.90.7, I hadn't tried the 1.0.0RC yet because the logstash website suggested I stick to 0.90.9 If you need any more logs / config samples I'd be happy to provide those.
Best, JG On Friday, January 31, 2014 3:46:07 PM UTC-5, David Montgomery wrote: > > Hi, > > In chef I used -N flag which changes the host name when using > chef-client. I removed the flag and built again. No difference. I have > two nodes running. Both have the name cluster name, ports are 100% wide > open...and wow..ES cant talk despite both running in the same region. > > So...ES...any suggestions? > > root@ip-10-238-131-236:/var/log/elasticsearch# hostname > ip-10-238-131-236 > > > root@ip-10-238-131-236:/var/log/elasticsearch# curl -XGET ' > http://127.0.0.1:9200/_cluster/health?pretty=true' > { > "cluster_name" : "mizilaelasticsearch", > "status" : "green", > "timed_out" : false, > "number_of_nodes" : 1, > "number_of_data_nodes" : 1, > "active_primary_shards" : 0, > "active_shards" : 0, > "relocating_shards" : 0, > "initializing_shards" : 0, > "unassigned_shards" : 0 > } > > > > > > > On Fri, Jan 31, 2014 at 12:20 AM, Ivan Brusic <iv...@brusic.com<javascript:> > > wrote: > >> Did you override your network.host configuration parameter by any chance? >> >> -- >> Ivan >> >> >> On Wed, Jan 29, 2014 at 1:56 PM, David Montgomery >> <davidmo...@gmail.com<javascript:> >> > wrote: >> >>> Hi, >>> >>> Details >>> >>> 1) both servers are sing the same sec group >>> 2 all ports between 9200 and 9300 are open 0.0.0.0/0 >>> 3) ec2 instacances in us-east region >>> 4) Did a fresh boot on 2 servers both servers are running. Both are not >>> talking >>> 5) Attached are the logs for both servers and the common >>> elasticsearch.yml file >>> 6) Below is he chef recipe I wrote. I am using 1.0.0.RC1 with >>> elasticsearch-cloud-aws/2.0.0.RC1 >>> /usr/share/elasticsearch/plugins/cloud-aws >>> ubuntu@ip-10-73-142-187:/usr/share/elasticsearch/plugins/cloud-aws$ ls >>> aws-java-sdk-1.3.32.jar commons-codec-1.4.jar >>> commons-logging-1.1.1.jar elasticsearch-cloud-aws-2.0.0.RC1.jar >>> httpclient-4.1.1.jar httpcore-4.1.jar >>> >>> >>> data_bag("my_data_bag") >>> db = data_bag_item("my_data_bag", "my") >>> AWS_ACCESS_KEY_ID = db[node.chef_environment]['aws']['AWS_ACCESS_KEY_ID'] >>> AWS_SECRET_ACCESS_KEY = >>> db[node.chef_environment]['aws']['AWS_SECRET_ACCESS_KEY'] >>> >>> >>> version = '1.0.0.RC1' >>> remote_file >>> "#{Chef::Config[:file_cache_path]}/elasticsearch-#{version}.deb" do >>> source " >>> https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-#{version}.deb >>> " >>> action :create_if_missing >>> end >>> >>> dpkg_package >>> "#{Chef::Config[:file_cache_path]}/elasticsearch-#{version}.deb" do >>> action :install >>> end >>> bash "add_aws" do >>> cwd "/usr/share/elasticsearch" >>> code <<-EOH >>> git clone >>> https://github.com/elasticsearch/elasticsearch-cloud-aws.git >>> /usr/share/elasticsearch/bin/plugin -install >>> elasticsearch/elasticsearch-cloud-aws/2.0.0.RC1 >>> EOH >>> not_if >>> {File.exists?("/usr/share/elasticsearch/elasticsearch-cloud-aws")} >>> end >>> >>> service "elasticsearch" do >>> supports :restart => true, :start => true, :stop => true >>> action [ :enable, :start] >>> end >>> >>> template "/etc/elasticsearch/logging.yml" do >>> path "/etc/elasticsearch/logging.yml" >>> source "logging.yml.erb" >>> owner "root" >>> group "root" >>> mode "0755" >>> notifies :restart, resources(:service => "elasticsearch") >>> end >>> >>> template "/etc/elasticsearch/elasticsearch.yml" do >>> path "/etc/elasticsearch/elasticsearch.yml" >>> source "elasticsearch.yml.erb" >>> owner "root" >>> group "root" >>> mode "0755" >>> variables :AWS_ACCESS_KEY_ID => "#{AWS_ACCESS_KEY_ID}", >>> :AWS_SECRET_ACCESS_KEY => "#{AWS_SECRET_ACCESS_KEY}", :region => "us-east-1" >>> notifies :restart, resources(:service => "elasticsearch") >>> end >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> On Tue, Jan 28, 2014 at 6:40 AM, David Montgomery >>> <davidmo...@gmail.com<javascript:> >>> > wrote: >>> >>>> sure..let me the logs together and will post two files ..one log file >>>> for each of the servers. >>>> >>>> Thanks >>>> >>>> >>>> On Tue, Jan 28, 2014 at 6:25 AM, David Pilato <da...@pilato.fr<javascript:> >>>> > wrote: >>>> >>>>> Is it a known problem? Not yet. That's the reason I'm asking for logs. >>>>> I would like to understand what is happening here. So full logs on both >>>>> nodes would help. >>>>> >>>>> The way you set unicast is fine to me. But please, could you share >>>>> with me (even through Direct mail) your elasticsearch.yml file and your >>>>> logs in TRACE level? >>>>> You can replace your credentials settings with XXXXXXXX. Please don't >>>>> change anything else. I'd like to reproduce your case. >>>>> >>>>> Thanks! >>>>> >>>>> >>>>> -- >>>>> David ;-) >>>>> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs >>>>> >>>>> >>>>> Le 27 janv. 2014 à 22:57, David Montgomery >>>>> <davidmo...@gmail.com<javascript:>> >>>>> a écrit : >>>>> >>>>> I have to say..it this a know problem? >>>>> >>>>> I removed the keys. >>>>> >>>>> So.. should the servers still cluster? Well....its not.........again >>>>> given that the two machine can talk to each other via telent, in the same >>>>> region, same sec group restarted services etc....no go......no cluster >>>>> >>>>> >>>>> So....how do I manually configure nodes? I have given up on auto >>>>> discover. Clearly does now work for me. >>>>> >>>>> Is the below an example? >>>>> >>>>> discovery.zen.ping.multicast.enabled: false >>>>> discovery.zen.ping.unicast.hosts: ["0.test.com:9200", "1.test.com >>>>> :9200"] >>>>> >>>>> >>>>> Thanks >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Tuesday, January 28, 2014 4:19:25 AM UTC+8, David Pilato wrote: >>>>>> >>>>>> Yes. Could you gist it (and remove first keys / secrets of course)? >>>>>> >>>>>> -- >>>>>> David ;-) >>>>>> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs >>>>>> >>>>>> >>>>>> Le 27 janv. 2014 à 20:53, David Montgomery <davidmo...@gmail.com> a >>>>>> écrit : >>>>>> >>>>>> I assume that means this in the logging.yml fike? >>>>>> # discovery >>>>>> discovery: TRACE >>>>>> >>>>>> I do have it on...what should I be looking for in the logs? >>>>>> >>>>>> On Tuesday, January 28, 2014 3:47:10 AM UTC+8, David Montgomery wrote: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> As an fyi..I can telnet into the other machine. Its only ES that >>>>>>> does not work despite following the online docs. >>>>>>> >>>>>>> If ES aws auto discovery does not work l assuming assume that >>>>>>> http://www.elasticsearch.org/guide/en/elasticsearch/ >>>>>>> reference/current/modules-discovery-zen.html will work? >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Tuesday, January 28, 2014 1:46:20 AM UTC+8, David Pilato wrote: >>>>>>>> >>>>>>>> Hi David, >>>>>>>> >>>>>>>> >>>>>>>> I think you should have nodes running under the same security >>>>>>>> group. In that case, you don't have to open 9300-9400 ports to public. >>>>>>>> If they don't belong to the same group or if you want to reach them >>>>>>>> from outside ec2 platform (public IP), then you need to set host_type >>>>>>>> to public_ip. >>>>>>>> >>>>>>>> It sounds like your nodes are trying to connect with private ip >>>>>>>> here (which is default). >>>>>>>> >>>>>>>> My 2 cents >>>>>>>> >>>>>>>> -- >>>>>>>> *David Pilato* | *Technical Advocate* | *Elasticsearch.com >>>>>>>> <http://Elasticsearch.com>* >>>>>>>> @dadoonet <https://twitter.com/dadoonet> | >>>>>>>> @elasticsearchfr<https://twitter.com/elasticsearchfr> >>>>>>>> >>>>>>>> >>>>>>>> Le 27 janvier 2014 at 18:07:48, David Montgomery ( >>>>>>>> davidmo...@gmail.com) a écrit: >>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I have two nodes runnging on ec2. >>>>>>>> >>>>>>>> curl -X GET http://localhost:9200/_cluster/health?pretty=true >>>>>>>> { >>>>>>>> "cluster_name" : "12345elasticsearch", >>>>>>>> "status" : "green", >>>>>>>> "timed_out" : false, >>>>>>>> "number_of_nodes" : 1, >>>>>>>> "number_of_data_nodes" : 1, >>>>>>>> "active_primary_shards" : 0, >>>>>>>> "active_shards" : 0, >>>>>>>> "relocating_shards" : 0, >>>>>>>> "initializing_shards" : 0, >>>>>>>> "unassigned_shards" : 0 >>>>>>>> } >>>>>>>> >>>>>>>> So..its fair to say that ec2 clustering did not work given the >>>>>>>> below config. Yes...the aws keys are correct and I have port >>>>>>>> 9200-9400 >>>>>>>> open. What elasticsearch advertiseres is that all that is needed. >>>>>>>> >>>>>>>> I am using 1.0.0.RC1 with elasticsearch-cloud-aws= 2.0.0.RC1 on >>>>>>>> ubuntu 12.04 >>>>>>>> >>>>>>>> >>>>>>>> cloud.aws.access_key: <%=@AWS_ACCESS_KEY_ID%> >>>>>>>> cloud.aws.secret_key: <%=@AWS_SECRET_ACCESS_KEY%> >>>>>>>> discovery.type: ec2 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> In the logs I see the below: >>>>>>>> >>>>>>>> [2014-01-27 16:58:49,708][TRACE][discovery.zen.ping.unicast] >>>>>>>> [Administrator] [1] failed to connect to [#cloud-i-1079ac6b-0][aws- >>>>>>>> elasticsearch-east-development-20140127160434][inet[/ >>>>>>>> 10.211.29.210:9300]] >>>>>>>> org.elasticsearch.transport.ConnectTransportException: [][inet[/ >>>>>>>> 10.211.29.210:9300]] connect_timeout[30s] >>>>>>>> at org.elasticsearch.transport.netty.NettyTransport. >>>>>>>> connectToChannelsLight(NettyTransport.java:676) >>>>>>>> at org.elasticsearch.transport.netty.NettyTransport. >>>>>>>> connectToNode(NettyTransport.java:636) >>>>>>>> at org.elasticsearch.transport.netty.NettyTransport. >>>>>>>> connectToNodeLight(NettyTransport.java:603) >>>>>>>> at org.elasticsearch.transport.TransportService. >>>>>>>> connectToNodeLight(TransportService.java:133) >>>>>>>> at org.elasticsearch.discovery.zen.ping.unicast. >>>>>>>> UnicastZenPing$3.run(UnicastZenPing.java:278) >>>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker( >>>>>>>> ThreadPoolExecutor.java:1146) >>>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>>>>>> ThreadPoolExecutor.java:615) >>>>>>>> at java.lang.Thread.run(Thread.java:701) >>>>>>>> Caused by: >>>>>>>> org.elasticsearch.common.netty.channel.ConnectTimeoutException: >>>>>>>> connection timed out: /10.239.36.98:9300 >>>>>>>> at org.elasticsearch.common.netty.channel.socket.nio. >>>>>>>> NioClientBoss.processConnectTimeout(NioClientBoss.java:137) >>>>>>>> at org.elasticsearch.common.netty.channel.socket.nio. >>>>>>>> NioClientBoss.process(NioClientBoss.java:83) >>>>>>>> at org.elasticsearch.common.netty.channel.socket.nio. >>>>>>>> AbstractNioSelector.run(AbstractNioSelector.java:318) >>>>>>>> at org.elasticsearch.common.netty.channel.socket.nio. >>>>>>>> NioClientBoss.run(NioClientBoss.java:42) >>>>>>>> at org.elasticsearch.common.netty.util. >>>>>>>> ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) >>>>>>>> at org.elasticsearch.common.netty.util.internal. >>>>>>>> DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) >>>>>>>> ... 3 more >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "elasticsearch" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to elasticsearc...@googlegroups.com. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/elasticsearch/ec028f53- >>>>>>>> c6b2-48e1-b714-a6f1ac7ae1cd%40googlegroups.com. >>>>>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>>>>> >>>>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "elasticsearch" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to elasticsearc...@googlegroups.com. >>>>>> To view this discussion on the web visit https://groups.google.com/d/ >>>>>> msgid/elasticsearch/4d1c068a-3e9c-4731-84e2-62eeeb63fa81% >>>>>> 40googlegroups.com. >>>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "elasticsearch" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to elasticsearc...@googlegroups.com <javascript:>. >>>>> >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/elasticsearch/2bdefbe0-3ab4-4b85-b473-8b502b812d5b%40googlegroups.com >>>>> . >>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>> >>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "elasticsearch" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/elasticsearch/PDDOu42M9Mc/unsubscribe >>>>> . >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> elasticsearc...@googlegroups.com <javascript:>. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/elasticsearch/2804A7F6-B2B8-4C23-B273-B7B7E0F805E0%40pilato.fr >>>>> . >>>>> >>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>> >>>> >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to elasticsearc...@googlegroups.com <javascript:>. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/CACF%2B8roDUwpq--dH9eAUoo-Xe-GEtgH%2B7WDJrbgjPWJBV2ijFQ%40mail.gmail.com >>> . >>> >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "elasticsearch" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/elasticsearch/PDDOu42M9Mc/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> elasticsearc...@googlegroups.com <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDHsDjF0AGUAoC3rLz_y5eGvFPtDEooRj0xP87mRjxurg%40mail.gmail.com >> . >> >> For more options, visit https://groups.google.com/groups/opt_out. >> > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a48fe555-0b95-4281-ba9e-351b21f4a9e9%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
