Yesterday I set the replica count to 0 with
curl -XPUT $(hostname -f):9200/_settings -d'{'index': {
'number_of_replicas':0}}'
and today the ES cluster assigned the new shards as it should.
I have now set the replica count back to 1 and will see if that's the
problem tomorrow.
On Tuesday, April 15, 2014 5:43:32 PM UTC+2, Andreas Paul wrote:
>
> Hi Mark,
>
> I forgot to write it again in this mail, but in the gist I pasted the full
> logs when the ES cluster created the new indices until I tried to restart
> the current active master.
>
> # head es_cluster.log
> [2014-04-14 02:00:01,504][INFO ][cluster.metadata ] [es@log01]
> [logstash-2014.04.14] creating index, cause [auto(bulk api)], shards [2]/[1],
> mappings [_default_]
> [2014-04-14 02:00:02,938][INFO ][cluster.metadata ] [es@log01]
> [puppetmaster-2014.04.14] creating index, cause [auto(bulk api)], shards
> [2]/[1], mappings []
> [2014-04-14 10:46:12,318][INFO ][node ] [es@log01]
> stopping ...
> [2014-04-14 10:46:12,446][WARN ][netty.channel.DefaultChannelPipeline] An
> exception was thrown by an exception handler.
> java.util.concurrent.RejectedExecutionException: Worker has already been
> shutdown
> at
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.registerTask(AbstractNioSelector.java:120)
> at
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:72)
>
>
>
>
> Which means that there are no log messages at all on why the cluster
> couldn't assign the new shards.
>
>
>
> On Tuesday, April 15, 2014 5:07:12 PM UTC+2, Mark Walkom wrote:
>>
>> Check your ES logs, there may be something there.
>>
>> Regards,
>> Mark Walkom
>>
>> Infrastructure Engineer
>> Campaign Monitor
>> email: ma...@campaignmonitor.com
>> web: www.campaignmonitor.com
>>
>>
>> On 15 April 2014 22:20, Andreas Paul <xor...@gmail.com> wrote:
>>
>>> Hello there,
>>>
>>> on Monday morning our ES cluster cluster switched to red, because he
>>> didn't assign the new created indices to any ES node, see attached picture.
>>>
>>>
>>>
>>> I tried manually allocating these unassigned shards to a node, but it
>>> only returned the following error:
>>>
>>> # curl -XPOST $(hostname -f):9200/_cluster/reroute?pretty=true -d
>>> '{"commands": [{"allocate": {"index": "foobar", "shard": 0, "node":
>>> "es@log09", "allow_primary": true }}]}'
>>>
>>> {
>>> "error" :
>>> "RemoteTransportException[[es@log05][inet[/12313.20.36.1337:9300]][cluster/reroute]];
>>> nested: IllegalFormatConversionException[d != java.lang.Double]; ",
>>> "status" : 400
>>>
>>> }
>>>
>>>
>>> Also see https://gist.github.com/xorpaul/10644099
>>>
>>> I also tried
>>>
>>> curl -XPUT $(hostname -f):9200/_settings -d '
>>> {"index.routing.allocation.disable_allocation": false}'
>>>
>>> and
>>>
>>> curl -XPUT $(hostname -f):9200/_settings -d '
>>> {"index.routing.allocation.enable": "all"}'
>>>
>>> and removing one node from the cluster, which seemed to help, because it
>>> finally assigned the shards to a node.
>>>
>>> Unfortunately the same problem appeared again the next day, when the
>>> cluster tried to create new indices.
>>>
>>> Elasticsearch 1.1.0 with OpenJDK Java7 on Debian Wheezy
>>>
>>>
>>> I would like to find out the reason why the cluster doesn't assign these
>>> new shards to any node or find a way to issue a command to the cluster to
>>> reassign/redistribute all unassigned shards to a node.
>>>
>>> Thanks in advance!
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/elasticsearch/c7b24957-7c97-4f2f-b1b1-70b61cb29669%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/c7b24957-7c97-4f2f-b1b1-70b61cb29669%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/9c825b3c-5556-46fd-9ed2-4c86df44f7ba%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
