Hi Mark,
I forgot to write it again in this mail, but in the gist I pasted the full
logs when the ES cluster created the new indices until I tried to restart
the current active master.
# head es_cluster.log
[2014-04-14 02:00:01,504][INFO ][cluster.metadata ] [es@log01]
[logstash-2014.04.14] creating index, cause [auto(bulk api)], shards [2]/[1],
mappings [_default_]
[2014-04-14 02:00:02,938][INFO ][cluster.metadata ] [es@log01]
[puppetmaster-2014.04.14] creating index, cause [auto(bulk api)], shards
[2]/[1], mappings []
[2014-04-14 10:46:12,318][INFO ][node ] [es@log01] stopping
...
[2014-04-14 10:46:12,446][WARN ][netty.channel.DefaultChannelPipeline] An
exception was thrown by an exception handler.
java.util.concurrent.RejectedExecutionException: Worker has already been
shutdown
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.registerTask(AbstractNioSelector.java:120)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:72)
Which means that there are no log messages at all on why the cluster
couldn't assign the new shards.
On Tuesday, April 15, 2014 5:07:12 PM UTC+2, Mark Walkom wrote:
>
> Check your ES logs, there may be something there.
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com <javascript:>
> web: www.campaignmonitor.com
>
>
> On 15 April 2014 22:20, Andreas Paul <xor...@gmail.com <javascript:>>wrote:
>
>> Hello there,
>>
>> on Monday morning our ES cluster cluster switched to red, because he
>> didn't assign the new created indices to any ES node, see attached picture.
>>
>>
>>
>> I tried manually allocating these unassigned shards to a node, but it
>> only returned the following error:
>>
>> # curl -XPOST $(hostname -f):9200/_cluster/reroute?pretty=true -d
>> '{"commands": [{"allocate": {"index": "foobar", "shard": 0, "node":
>> "es@log09", "allow_primary": true }}]}'
>>
>> {
>> "error" :
>> "RemoteTransportException[[es@log05][inet[/12313.20.36.1337:9300]][cluster/reroute]];
>> nested: IllegalFormatConversionException[d != java.lang.Double]; ",
>> "status" : 400
>>
>> }
>>
>>
>> Also see https://gist.github.com/xorpaul/10644099
>>
>> I also tried
>>
>> curl -XPUT $(hostname -f):9200/_settings -d '
>> {"index.routing.allocation.disable_allocation": false}'
>>
>> and
>>
>> curl -XPUT $(hostname -f):9200/_settings -d '
>> {"index.routing.allocation.enable": "all"}'
>>
>> and removing one node from the cluster, which seemed to help, because it
>> finally assigned the shards to a node.
>>
>> Unfortunately the same problem appeared again the next day, when the
>> cluster tried to create new indices.
>>
>> Elasticsearch 1.1.0 with OpenJDK Java7 on Debian Wheezy
>>
>>
>> I would like to find out the reason why the cluster doesn't assign these
>> new shards to any node or find a way to issue a command to the cluster to
>> reassign/redistribute all unassigned shards to a node.
>>
>> Thanks in advance!
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearc...@googlegroups.com <javascript:>.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/c7b24957-7c97-4f2f-b1b1-70b61cb29669%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/c7b24957-7c97-4f2f-b1b1-70b61cb29669%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2cb5fb25-8687-47a0-bb98-40b7a01074c7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
