Hi Jan,
Upgrading to 2.x version will take at-least 6-7 months for us to
migrate. About the heap, it is at 70% and no issues with it. No logs are
showing up based on the slow rate, apart from what i have posted. There was
an error on indice which mentioned about not able to calculate the range, i
did the "Recalculate index cycles" which fixed that, but the slow
processing has accumulated about 100,000,000 messages in the journal on the
Master Node.
Still finding what is happening.. :-(
Thanks Hema.
On Friday, July 8, 2016 at 5:40:47 PM UTC+5:30, Hema Kumar wrote:
>
> Hi,
>I am using graylog 1.3.3 with ES 1.7.5, from yesterday we are seeing
> the process buffer filled up on the master node and the outgoing process is
> too slow than normal, I have tried restarting GL and ES but did not fix the
> issue, below are the log warn and errors we see that repeats continuously.
>
> We have 4 graylog server and 7 elasticsearch nodes, Only the Master
> graylog is processing slow and sometimes the 3rd node, rest of the nodes
> are working fine.
>
> Could you please help me on this, i have been breaking my head since
> yesterday.
>
>
> 2016-07-08T01:53:21.355-06:00 WARN [GelfChunkAggregator] Error while
> expiring GELF chunk entries
> java.util.NoSuchElementException
> at
> java.util.concurrent.ConcurrentSkipListMap.firstKey(ConcurrentSkipListMap.java:2036)
> at
> java.util.concurrent.ConcurrentSkipListSet.first(ConcurrentSkipListSet.java:396)
> at
> org.graylog2.inputs.codecs.GelfChunkAggregator$ChunkEvictionTask.run(GelfChunkAggregator.java:288)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> 2016-07-08T05:37:47.803-06:00 ERROR [AnyExceptionClassMapper] Unhandled
> exception in REST resource
> org.elasticsearch.action.search.ReduceSearchPhaseException: Failed to
> execute phase [fetch], [reduce]
> at
> org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$2.onFailure(TransportSearchQueryThenFetchAction.java:159)
> at
> org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:41)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ClassCastException:
> org.elasticsearch.search.aggregations.bucket.terms.LongTerms$Bucket cannot
> be cast to
> org.elasticsearch.search.aggregations.bucket.terms.StringTerms$Bucket
> at
> org.elasticsearch.search.aggregations.bucket.terms.StringTerms$Bucket.compareTerm(StringTerms.java:85)
> at
> org.elasticsearch.search.aggregations.bucket.terms.InternalOrder$4.compare(InternalOrder.java:87)
> at
> org.elasticsearch.search.aggregations.bucket.terms.InternalOrder$4.compare(InternalOrder.java:83)
> at
> org.elasticsearch.search.aggregations.bucket.terms.InternalOrder$CompoundOrder$CompoundOrderComparator.compare(InternalOrder.java:284)
> at
> org.elasticsearch.search.aggregations.bucket.terms.InternalOrder$CompoundOrder$CompoundOrderComparator.compare(InternalOrder.java:270)
> at
> org.elasticsearch.search.aggregations.bucket.terms.support.BucketPriorityQueue.lessThan(BucketPriorityQueue.java:37)
> at
> org.elasticsearch.search.aggregations.bucket.terms.support.BucketPriorityQueue.lessThan(BucketPriorityQueue.java:26)
> at
> org.apache.lucene.util.PriorityQueue.upHeap(PriorityQueue.java:225)
> at org.apache.lucene.util.PriorityQueue.add(PriorityQueue.java:133)
> at
> org.apache.lucene.util.PriorityQueue.insertWithOverflow(PriorityQueue.java:149)
> at
> org.elasticsearch.search.aggregations.bucket.terms.InternalTerms.reduce(InternalTerms.java:195)
> at
> org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:140)
> at
> org.elasticsearch.search.aggregations.bucket.InternalSingleBucketAggregation.reduce(InternalSingleBucketAggregation.java:79)
> at
> org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:140)
> at
> org.elasticsearch.search.controller.SearchPhaseController.merge(SearchPhaseController.java:407)
> at
> org.elasti
