Today the ES cluster still works as expected.
Still don't know the reason why it failed in the first place or what I did
to fix it.
Maybe a slow cluster restart helped: stopping all nodes and then starting
only one node so it can become master instead of restarting all at once and
letting them work it out who should become master.
Maybe I had a split brain problem by restarting too quickly, but then why
would I see all nodes in the cluster information with only one master.
Anyway it's working now...
On Wednesday, April 16, 2014 10:26:15 AM UTC+2, Andreas Paul wrote:
>
> Yesterday I set the replica count to 0 with
>
> curl -XPUT $(hostname -f):9200/_settings -d'{'index': {
> 'number_of_replicas':0}}'
>
> and today the ES cluster assigned the new shards as it should.
>
> I have now set the replica count back to 1 and will see if that's the
> problem tomorrow.
>
>
> On Tuesday, April 15, 2014 5:43:32 PM UTC+2, Andreas Paul wrote:
>>
>> Hi Mark,
>>
>> I forgot to write it again in this mail, but in the gist I pasted the
>> full logs when the ES cluster created the new indices until I tried to
>> restart the current active master.
>>
>> # head es_cluster.log
>> [2014-04-14 02:00:01,504][INFO ][cluster.metadata ] [es@log01]
>> [logstash-2014.04.14] creating index, cause [auto(bulk api)], shards
>> [2]/[1], mappings [_default_]
>> [2014-04-14 02:00:02,938][INFO ][cluster.metadata ] [es@log01]
>> [puppetmaster-2014.04.14] creating index, cause [auto(bulk api)], shards
>> [2]/[1], mappings []
>> [2014-04-14 10:46:12,318][INFO ][node ] [es@log01]
>> stopping ...
>> [2014-04-14 10:46:12,446][WARN ][netty.channel.DefaultChannelPipeline] An
>> exception was thrown by an exception handler.
>> java.util.concurrent.RejectedExecutionException: Worker has already been
>> shutdown
>> at
>> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.registerTask(AbstractNioSelector.java:120)
>> at
>> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:72)
>>
>>
>>
>>
>> Which means that there are no log messages at all on why the cluster
>> couldn't assign the new shards.
>>
>>
>>
>> On Tuesday, April 15, 2014 5:07:12 PM UTC+2, Mark Walkom wrote:
>>>
>>> Check your ES logs, there may be something there.
>>>
>>> Regards,
>>> Mark Walkom
>>>
>>> Infrastructure Engineer
>>> Campaign Monitor
>>> email: ma...@campaignmonitor.com
>>> web: www.campaignmonitor.com
>>>
>>>
>>> On 15 April 2014 22:20, Andreas Paul <xor...@gmail.com> wrote:
>>>
>>>> Hello there,
>>>>
>>>> on Monday morning our ES cluster cluster switched to red, because he
>>>> didn't assign the new created indices to any ES node, see attached picture.
>>>>
>>>>
>>>>
>>>> I tried manually allocating these unassigned shards to a node, but it
>>>> only returned the following error:
>>>>
>>>> # curl -XPOST $(hostname -f):9200/_cluster/reroute?pretty=true -d
>>>> '{"commands": [{"allocate": {"index": "foobar", "shard": 0, "node":
>>>> "es@log09", "allow_primary": true }}]}'
>>>>
>>>> {
>>>> "error" :
>>>> "RemoteTransportException[[es@log05][inet[/12313.20.36.1337:9300]][cluster/reroute]];
>>>> nested: IllegalFormatConversionException[d != java.lang.Double]; ",
>>>> "status" : 400
>>>>
>>>> }
>>>>
>>>>
>>>> Also see https://gist.github.com/xorpaul/10644099
>>>>
>>>> I also tried
>>>>
>>>> curl -XPUT $(hostname -f):9200/_settings -d '
>>>> {"index.routing.allocation.disable_allocation": false}'
>>>>
>>>> and
>>>>
>>>> curl -XPUT $(hostname -f):9200/_settings -d '
>>>> {"index.routing.allocation.enable": "all"}'
>>>>
>>>> and removing one node from the cluster, which seemed to help, because
>>>> it finally assigned the shards to a node.
>>>>
>>>> Unfortunately the same problem appeared again the next day, when the
>>>> cluster tried to create new indices.
>>>>
>>>> Elasticsearch 1.1.0 with OpenJDK Java7 on Debian Wheezy
>>>>
>>>>
>>>> I would like to find out the reason why the cluster doesn't assign
>>>> these new shards to any node or find a way to issue a command to the
>>>> cluster to reassign/redistribute all unassigned shards to a node.
>>>>
>>>> Thanks in advance!
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "elasticsearch" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to elasticsearc...@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/elasticsearch/c7b24957-7c97-4f2f-b1b1-70b61cb29669%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/c7b24957-7c97-4f2f-b1b1-70b61cb29669%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/821de798-ba3e-4d36-9373-292344a95217%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
