Re: ES cluster unable to assing new shards
Today the ES cluster still works as expected.
Still don't know the reason why it failed in the first place or what I did
to fix it.
Maybe a slow cluster restart helped: stopping all nodes and then starting
only one node so it can become master instead of restarting all at once and
letting them work it out who should become master.
Maybe I had a split brain problem by restarting too quickly, but then why
would I see all nodes in the cluster information with only one master.
Anyway it's working now...
On Wednesday, April 16, 2014 10:26:15 AM UTC+2, Andreas Paul wrote:
>
> Yesterday I set the replica count to 0 with
>
> curl -XPUT $(hostname -f):9200/_settings -d'{'index': {
> 'number_of_replicas':0}}'
>
> and today the ES cluster assigned the new shards as it should.
>
> I have now set the replica count back to 1 and will see if that's the
> problem tomorrow.
>
>
> On Tuesday, April 15, 2014 5:43:32 PM UTC+2, Andreas Paul wrote:
>>
>> Hi Mark,
>>
>> I forgot to write it again in this mail, but in the gist I pasted the
>> full logs when the ES cluster created the new indices until I tried to
>> restart the current active master.
>>
>> # head es_cluster.log
>> [2014-04-14 02:00:01,504][INFO ][cluster.metadata ] [es@log01]
>> [logstash-2014.04.14] creating index, cause [auto(bulk api)], shards
>> [2]/[1], mappings [_default_]
>> [2014-04-14 02:00:02,938][INFO ][cluster.metadata ] [es@log01]
>> [puppetmaster-2014.04.14] creating index, cause [auto(bulk api)], shards
>> [2]/[1], mappings []
>> [2014-04-14 10:46:12,318][INFO ][node ] [es@log01]
>> stopping ...
>> [2014-04-14 10:46:12,446][WARN ][netty.channel.DefaultChannelPipeline] An
>> exception was thrown by an exception handler.
>> java.util.concurrent.RejectedExecutionException: Worker has already been
>> shutdown
>> at
>> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.registerTask(AbstractNioSelector.java:120)
>> at
>> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:72)
>>
>>
>>
>>
>> Which means that there are no log messages at all on why the cluster
>> couldn't assign the new shards.
>>
>>
>>
>> On Tuesday, April 15, 2014 5:07:12 PM UTC+2, Mark Walkom wrote:
>>>
>>> Check your ES logs, there may be something there.
>>>
>>> Regards,
>>> Mark Walkom
>>>
>>> Infrastructure Engineer
>>> Campaign Monitor
>>> email: ma...@campaignmonitor.com
>>> web: www.campaignmonitor.com
>>>
>>>
>>> On 15 April 2014 22:20, Andreas Paul wrote:
>>>
Hello there,
on Monday morning our ES cluster cluster switched to red, because he
didn't assign the new created indices to any ES node, see attached picture.
I tried manually allocating these unassigned shards to a node, but it
only returned the following error:
# curl -XPOST $(hostname -f):9200/_cluster/reroute?pretty=true -d
'{"commands": [{"allocate": {"index": "foobar", "shard": 0, "node":
"es@log09", "allow_primary": true }}]}'
{
"error" :
"RemoteTransportException[[es@log05][inet[/12313.20.36.1337:9300]][cluster/reroute]];
nested: IllegalFormatConversionException[d != java.lang.Double]; ",
"status" : 400
}
Also see https://gist.github.com/xorpaul/10644099
I also tried
curl -XPUT $(hostname -f):9200/_settings -d '
{"index.routing.allocation.disable_allocation": false}'
and
curl -XPUT $(hostname -f):9200/_settings -d '
{"index.routing.allocation.enable": "all"}'
and removing one node from the cluster, which seemed to help, because
it finally assigned the shards to a node.
Unfortunately the same problem appeared again the next day, when the
cluster tried to create new indices.
Elasticsearch 1.1.0 with OpenJDK Java7 on Debian Wheezy
I would like to find out the reason why the cluster doesn't assign
these new shards to any node or find a way to issue a command to the
cluster to reassign/redistribute all unassigned shards to a node.
Thanks in advance!
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c7b24957-7c97-4f2f-b1b1-70b61cb29669%40googlegroups.com
.
For more options, visit https://groups.google.com/d/optout.
>>>
>>>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receivin
Re: ES cluster unable to assing new shards
Yesterday I set the replica count to 0 with
curl -XPUT $(hostname -f):9200/_settings -d'{'index': {
'number_of_replicas':0}}'
and today the ES cluster assigned the new shards as it should.
I have now set the replica count back to 1 and will see if that's the
problem tomorrow.
On Tuesday, April 15, 2014 5:43:32 PM UTC+2, Andreas Paul wrote:
>
> Hi Mark,
>
> I forgot to write it again in this mail, but in the gist I pasted the full
> logs when the ES cluster created the new indices until I tried to restart
> the current active master.
>
> # head es_cluster.log
> [2014-04-14 02:00:01,504][INFO ][cluster.metadata ] [es@log01]
> [logstash-2014.04.14] creating index, cause [auto(bulk api)], shards [2]/[1],
> mappings [_default_]
> [2014-04-14 02:00:02,938][INFO ][cluster.metadata ] [es@log01]
> [puppetmaster-2014.04.14] creating index, cause [auto(bulk api)], shards
> [2]/[1], mappings []
> [2014-04-14 10:46:12,318][INFO ][node ] [es@log01]
> stopping ...
> [2014-04-14 10:46:12,446][WARN ][netty.channel.DefaultChannelPipeline] An
> exception was thrown by an exception handler.
> java.util.concurrent.RejectedExecutionException: Worker has already been
> shutdown
> at
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.registerTask(AbstractNioSelector.java:120)
> at
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:72)
>
>
>
>
> Which means that there are no log messages at all on why the cluster
> couldn't assign the new shards.
>
>
>
> On Tuesday, April 15, 2014 5:07:12 PM UTC+2, Mark Walkom wrote:
>>
>> Check your ES logs, there may be something there.
>>
>> Regards,
>> Mark Walkom
>>
>> Infrastructure Engineer
>> Campaign Monitor
>> email: ma...@campaignmonitor.com
>> web: www.campaignmonitor.com
>>
>>
>> On 15 April 2014 22:20, Andreas Paul wrote:
>>
>>> Hello there,
>>>
>>> on Monday morning our ES cluster cluster switched to red, because he
>>> didn't assign the new created indices to any ES node, see attached picture.
>>>
>>>
>>>
>>> I tried manually allocating these unassigned shards to a node, but it
>>> only returned the following error:
>>>
>>> # curl -XPOST $(hostname -f):9200/_cluster/reroute?pretty=true -d
>>> '{"commands": [{"allocate": {"index": "foobar", "shard": 0, "node":
>>> "es@log09", "allow_primary": true }}]}'
>>>
>>> {
>>> "error" :
>>> "RemoteTransportException[[es@log05][inet[/12313.20.36.1337:9300]][cluster/reroute]];
>>> nested: IllegalFormatConversionException[d != java.lang.Double]; ",
>>> "status" : 400
>>>
>>> }
>>>
>>>
>>> Also see https://gist.github.com/xorpaul/10644099
>>>
>>> I also tried
>>>
>>> curl -XPUT $(hostname -f):9200/_settings -d '
>>> {"index.routing.allocation.disable_allocation": false}'
>>>
>>> and
>>>
>>> curl -XPUT $(hostname -f):9200/_settings -d '
>>> {"index.routing.allocation.enable": "all"}'
>>>
>>> and removing one node from the cluster, which seemed to help, because it
>>> finally assigned the shards to a node.
>>>
>>> Unfortunately the same problem appeared again the next day, when the
>>> cluster tried to create new indices.
>>>
>>> Elasticsearch 1.1.0 with OpenJDK Java7 on Debian Wheezy
>>>
>>>
>>> I would like to find out the reason why the cluster doesn't assign these
>>> new shards to any node or find a way to issue a command to the cluster to
>>> reassign/redistribute all unassigned shards to a node.
>>>
>>> Thanks in advance!
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/elasticsearch/c7b24957-7c97-4f2f-b1b1-70b61cb29669%40googlegroups.com
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/9c825b3c-5556-46fd-9ed2-4c86df44f7ba%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: ES cluster unable to assing new shards
Hi Mark,
I forgot to write it again in this mail, but in the gist I pasted the full
logs when the ES cluster created the new indices until I tried to restart
the current active master.
# head es_cluster.log
[2014-04-14 02:00:01,504][INFO ][cluster.metadata ] [es@log01]
[logstash-2014.04.14] creating index, cause [auto(bulk api)], shards [2]/[1],
mappings [_default_]
[2014-04-14 02:00:02,938][INFO ][cluster.metadata ] [es@log01]
[puppetmaster-2014.04.14] creating index, cause [auto(bulk api)], shards
[2]/[1], mappings []
[2014-04-14 10:46:12,318][INFO ][node ] [es@log01] stopping
...
[2014-04-14 10:46:12,446][WARN ][netty.channel.DefaultChannelPipeline] An
exception was thrown by an exception handler.
java.util.concurrent.RejectedExecutionException: Worker has already been
shutdown
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.registerTask(AbstractNioSelector.java:120)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:72)
Which means that there are no log messages at all on why the cluster
couldn't assign the new shards.
On Tuesday, April 15, 2014 5:07:12 PM UTC+2, Mark Walkom wrote:
>
> Check your ES logs, there may be something there.
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com
> web: www.campaignmonitor.com
>
>
> On 15 April 2014 22:20, Andreas Paul >wrote:
>
>> Hello there,
>>
>> on Monday morning our ES cluster cluster switched to red, because he
>> didn't assign the new created indices to any ES node, see attached picture.
>>
>>
>>
>> I tried manually allocating these unassigned shards to a node, but it
>> only returned the following error:
>>
>> # curl -XPOST $(hostname -f):9200/_cluster/reroute?pretty=true -d
>> '{"commands": [{"allocate": {"index": "foobar", "shard": 0, "node":
>> "es@log09", "allow_primary": true }}]}'
>>
>> {
>> "error" :
>> "RemoteTransportException[[es@log05][inet[/12313.20.36.1337:9300]][cluster/reroute]];
>> nested: IllegalFormatConversionException[d != java.lang.Double]; ",
>> "status" : 400
>>
>> }
>>
>>
>> Also see https://gist.github.com/xorpaul/10644099
>>
>> I also tried
>>
>> curl -XPUT $(hostname -f):9200/_settings -d '
>> {"index.routing.allocation.disable_allocation": false}'
>>
>> and
>>
>> curl -XPUT $(hostname -f):9200/_settings -d '
>> {"index.routing.allocation.enable": "all"}'
>>
>> and removing one node from the cluster, which seemed to help, because it
>> finally assigned the shards to a node.
>>
>> Unfortunately the same problem appeared again the next day, when the
>> cluster tried to create new indices.
>>
>> Elasticsearch 1.1.0 with OpenJDK Java7 on Debian Wheezy
>>
>>
>> I would like to find out the reason why the cluster doesn't assign these
>> new shards to any node or find a way to issue a command to the cluster to
>> reassign/redistribute all unassigned shards to a node.
>>
>> Thanks in advance!
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/c7b24957-7c97-4f2f-b1b1-70b61cb29669%40googlegroups.com
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2cb5fb25-8687-47a0-bb98-40b7a01074c7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: ES cluster unable to assing new shards
Check your ES logs, there may be something there.
Regards,
Mark Walkom
Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com
On 15 April 2014 22:20, Andreas Paul wrote:
> Hello there,
>
> on Monday morning our ES cluster cluster switched to red, because he
> didn't assign the new created indices to any ES node, see attached picture.
>
>
>
> I tried manually allocating these unassigned shards to a node, but it only
> returned the following error:
>
> # curl -XPOST $(hostname -f):9200/_cluster/reroute?pretty=true -d
> '{"commands": [{"allocate": {"index": "foobar", "shard": 0, "node":
> "es@log09", "allow_primary": true }}]}'
> {
> "error" :
> "RemoteTransportException[[es@log05][inet[/12313.20.36.1337:9300]][cluster/reroute]];
> nested: IllegalFormatConversionException[d != java.lang.Double]; ",
> "status" : 400
> }
>
>
> Also see https://gist.github.com/xorpaul/10644099
>
> I also tried
>
> curl -XPUT $(hostname -f):9200/_settings -d '
> {"index.routing.allocation.disable_allocation": false}'
>
> and
>
> curl -XPUT $(hostname -f):9200/_settings -d '
> {"index.routing.allocation.enable": "all"}'
>
> and removing one node from the cluster, which seemed to help, because it
> finally assigned the shards to a node.
>
> Unfortunately the same problem appeared again the next day, when the
> cluster tried to create new indices.
>
> Elasticsearch 1.1.0 with OpenJDK Java7 on Debian Wheezy
>
>
> I would like to find out the reason why the cluster doesn't assign these
> new shards to any node or find a way to issue a command to the cluster to
> reassign/redistribute all unassigned shards to a node.
>
> Thanks in advance!
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/c7b24957-7c97-4f2f-b1b1-70b61cb29669%40googlegroups.com
> .
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624bko7rK%3Ds%2Bp9oJFdCW9VdykCusUP0DJER8%3Dj%3Dj9%2BtLLBQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
