Hi, we'd been using ES for a while now. Specifically version 0.90.3. A
couple of months ago we decided to migrate to the latest version which was
finally frozen to be 1.4.1. No data migration was necessary because we have
a redundant MongoDB, but yesterday we enabled data writing to the new ES
cluster. All was running smoothly when we noticed that at o'clock times
there were bursts of four or five log messages of the following kinds:
Error indexing None into index ind-analytics-2015.01.08. Total elapsed
time: 1065 ms.
org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException:
failed to process cluster event (acquire index lock) within 1s
at
org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$1.run(MetaDataCreateIndexService.java:148)
~[org.elasticsearch.elasticsearch-1.4.1.jar:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_17]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_17]
at java.lang.Thread.run(Thread.java:722) ~[na:1.7.0_17]
[ForkJoinPool-2-worker-15] c.d.i.p.ActorScatterGatherStrategy - Scattering
to failed in 1043ms org.elasticsearch.action.UnavailableShardsException:
[ind-2015.01.08.00][0] Not enough active copies to meet write consistency
of [QUORUM] (have 1, needed 2). Timeout: [1s], request: index
{[ind-2015.01.08.00][search][...]}
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.retryBecauseUnavailable(TransportShardReplicationOperationAction.java:784)
~[org.elasticsearch.elasticsearch-1.4.1.jar:na]
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.raiseFailureIfHaveNotEnoughActiveShardCopies(TransportShardReplicationOperationAction.java:776)
~[org.elasticsearch.elasticsearch-1.4.1.jar:na]
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:507)
~[org.elasticsearch.elasticsearch-1.4.1.jar:na]
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:419)
~[org.elasticsearch.elasticsearch-1.4.1.jar:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_17]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_17]
at java.lang.Thread.run(Thread.java:722) ~[na:1.7.0_17]
This occurs at o'clock times because we write over hour-based indices. For
example, all writes from 18:00:00 to 18:59:59 of 01/08 goes to
ind-2015.01.08.18. At 19:00:00 all writes will go to ind-2015.01.08.19, and
so on.
With 0.90.3 version of ES, automatic index creation was working flawlessly
(with no complaints) but the new version doesn't seem to handle that
feature very well. It looks like, when all those concurrent writes competes
to be the first to create the index, all but one fails. Of course we could
just create such indices manually to avoid this situation altogether, but
this would only be a workaround for a feature that previously worked.
Also, we use ES through the native Java client and the configuration for
all our indices is
settings = {
number_of_shards = 5,
number_of_replicas = 2
}
Any ideas?
Thanks in advance,
Tom;
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/4deefb09-bed1-499a-b9fc-3ed4d78fc4c0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
