[graylog2] graylog stops outputting messages to elasticsearch every few days

[remotecontrol]

Hello,

We recently set up a new host with graylog 1.2.2 and elasticsearch 1.7.3 
from the centos repos available. Every 3 or 4 days graylog will lose it's 
connection to elasticsearch and stop outputting messages to the database. 
The messages are kept in the disk buffer and will be put into elasticsearch 
upon restarting the graylog-server process. 

The graylog web interface reports everything is green and curl reqests to 
elasticsearch show a-ok. I found the following message in the 
graylog-server log but it does not say why... 

2015-11-24T08:52:37.389-08:00 WARN  [BlockingBatchedESOutput] Error while 
waiting for healthy Elasticsearch cluster. Not flushing.

java.util.concurrent.TimeoutException: Elasticsearch cluster didn't get 
healthy within timeout

at 
org.graylog2.indexer.cluster.Cluster.waitForConnectedAndHealthy(Cluster.java:174)

at 
org.graylog2.indexer.cluster.Cluster.waitForConnectedAndHealthy(Cluster.java:179)

at 
org.graylog2.outputs.BlockingBatchedESOutput.flush(BlockingBatchedESOutput.java:112)

at 
org.graylog2.outputs.BlockingBatchedESOutput.write(BlockingBatchedESOutput.java:105)

at 
org.graylog2.buffers.processors.OutputBufferProcessor$1.run(OutputBufferProcessor.java:189)

at 
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)


Our ES is configured to use 30G of ram and graylog server is configured for 
4GB. The host has 74GB total.


Let me know if i can provide any more info. Anyone experience this? Our old 
setup did not have this problem...


Thanks,

Josh

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/ad0c172a-cfb5-493f-bd71-3eb021c7a51a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Change permission for user

[Hitsu Yaga]

Hi Anant,
I install graylog through repo and yum. It's possible to disable the System 
menu? If I want admin can access System menu and user can not, how can I 
modify it? Thanks for your help!!!

On Thursday, November 26, 2015 at 5:15:22 PM UTC+7, Anant Sawant wrote:
>
> Hi!!
>
> If you are running your own compiled source code for Graylog and not the 
> ready to run Graylog package you can simply hide this menu for end user.
> Just disable the System menu from app.js file located in the 
> "graylog-web-interface.graylog-web-interface-1.1.6-assets" jar. Or under 
> source code.
>
> Cheers,
> Anant.
>
> On Thursday, 26 November 2015 09:34:36 UTC+5:30, Hitsu Yaga wrote:
>>
>> Dear all,
>> I am also facing with this problem now. I don't want customer see 
>> anything about our node, systemHow can I do that? please advise!!!
>>
>> On Wednesday, November 25, 2015 at 11:15:02 PM UTC+7, Tiểu Yết wrote:
>>>
>>> Hi everyone,
>>> We just setup graylog server for our customers can access their logs. 
>>> Now we can do it with stream that seperate log with specific field. But 
>>> when our customer login web-interface they can access something about our 
>>> system as attached file. We only want they only access stream and 
>>> dashboard. It's possible?
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/c93367a6-3bd3-493e-8ba0-48af7f3f9c08%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] journal broken

[teemu]

Thanks! New journal directory helped us also. Do you know how to fix and 
process those old journal files?

On Monday, March 2, 2015 at 6:10:49 PM UTC+2, Ed Totman wrote:
>
> I deleted the journal and re-enabled it, and also changed 
> the index.refresh_interval as recommended by Tristan.
>
> On Monday, March 2, 2015 at 3:05:10 AM UTC-8, Bernd Ahlers wrote:
>>
>> Ed, 
>>
>> if you want to delete all of the journal, stop the server, delete the 
>> journal dir (see "message_journal_dir" setting in graylog.conf) and 
>> start the server again. 
>>
>> Bernd 
>>
>> On 26 February 2015 at 16:13, Ed Totman  wrote: 
>> > Thanks for the reply.  How do I clear the journal of old messages 
>> before I 
>> > restart it? 
>> > 
>> > On Wednesday, February 25, 2015 at 10:54:42 PM UTC-8, Bernd Ahlers 
>> wrote: 
>> >> 
>> >> Ed, 
>> >> 
>> >> as Tristan already said, if you constantly sending in more messages 
>> >> than Graylog or Elasticsearch can process, you will always fill up 
>> >> your journal. 
>> >> Disabling the journal does not really fix the problem, because you 
>> >> will now lose messages. 
>> >> 
>> >> Please check the node details page (System -> Nodes -> click on the 
>> >> node name) and check the disk journal stats. If you writing more into 
>> >> the journal than reading from it, you have a problem with processing 
>> >> throughput. 
>> >> 
>> >> Regards, 
>> >> Bernd 
>> >> 
>> >> On 26 February 2015 at 00:50, Tristan Rhodes  
>> wrote: 
>> >> > Ed, 
>> >> > 
>> >> > I had this same problem.  However, increasing the journal size will 
>> only 
>> >> > help if your rate of messages periodically decreases below what your 
>> >> > system 
>> >> > can process.  (For example, you will grow the journal during peak 
>> hours 
>> >> > of 
>> >> > the day, and drain the journal when fewer logs are being sent to 
>> >> > Graylog). 
>> >> > 
>> >> > If you are always sending more messages than your Elasticsearch can 
>> >> > ingest, 
>> >> > the journal will not help.  I increased my Elasticsearch ingesting 
>> >> > performance by changing this setting in elasticsearch.yml: 
>> >> > 
>> >> > index.refresh_interval: 30s 
>> >> > 
>> >> > You can read more about this setting here: 
>> >> > 
>> >> > 
>> >> > 
>> http://blog.sematext.com/2013/07/08/elasticsearch-refresh-interval-vs-indexing-performance/
>>  
>> >> > 
>> >> > 
>> http://www.elasticsearch.org/blog/performance-considerations-elasticsearch-indexing/
>>  
>> >> > 
>> >> > Disclaimer: I am new to graylog+elastisearch and barely know what I 
>> am 
>> >> > doing.  :) 
>> >> > 
>> >> > Cheers! 
>> >> > 
>> >> > Tristan 
>> >> > 
>> >> > On Mon, Feb 23, 2015 at 10:41 AM, Ed Totman  
>> wrote: 
>> >> >> 
>> >> >> I deployed the latest appliance from the ova file.  Graylog2 worked 
>> >> >> fine 
>> >> >> for several days, but then the journal files grew to 5GB which is 
>> the 
>> >> >> default limit and search returns no current results.  On the System 
>> >> >> page 
>> >> >> this error appeared: 
>> >> >> 
>> >> >> Journal utilization is too high a few seconds ago 
>> >> >> Journal utilization is too high and may go over the limit soon. 
>> Please 
>> >> >> verify that your Elasticsearch cluster is healthy and fast enough. 
>> You 
>> >> >> may 
>> >> >> also want to review your Graylog journal settings and set a higher 
>> >> >> limit. 
>> >> >> (Node: 43a9cc82-dc5a-4492-936b-418e1bc98f5e, journal utilization: 
>> >> >> 96.0%) 
>> >> >> 
>> >> >> I increased the journal limit to 10GB but this did not fix the 
>> problem. 
>> >> >> I 
>> >> >> restarted all services and checked the logs, but could not find any 
>> >> >> obvious 
>> >> >> problem.  The VM is running on very fast storage with lots of CPU 
>> and 
>> >> >> memory.  I set "message_journal_enabled = false" which seems to 
>> have 
>> >> >> temporarily resolved the problem. 
>> >> >> 
>> >> >> How do I troubleshoot the journal?  All of the other components are 
>> >> >> working fine. 
>> >> >> 
>> >> >> -- 
>> >> >> You received this message because you are subscribed to the Google 
>> >> >> Groups 
>> >> >> "graylog2" group. 
>> >> >> To unsubscribe from this group and stop receiving emails from it, 
>> send 
>> >> >> an 
>> >> >> email to graylog2+u...@googlegroups.com. 
>> >> >> For more options, visit https://groups.google.com/d/optout. 
>> >> > 
>> >> > 
>> >> > 
>> >> > 
>> >> > -- 
>> >> > Tristan Rhodes 
>> >> > 
>> >> > -- 
>> >> > You received this message because you are subscribed to the Google 
>> >> > Groups 
>> >> > "graylog2" group. 
>> >> > To unsubscribe from this group and stop receiving emails from it, 
>> send 
>> >> > an 
>> >> > email to graylog2+u...@googlegroups.com. 
>> >> > For more options, visit https://groups.google.com/d/optout. 
>> >> 
>> >> 
>> >> 
>> >> -- 
>> >> Developer 
>> >> 
>> >> Tel.: +49 (0)40 609 452 077 
>> >> Fax.: +49 (0)40 609 452 078 
>> >> 
>> >> TORCH GmbH - A Graylog company 
>> >> 

[graylog2] Graylog dashboard not showing the sources list

[Sriranga Kulkarni]

Hi,

Often i see my graylog dashboard sources tab doesnt work. I had to restart 
the graylog server and web and elasticsearch then i am able to see the 
sources list. It happens very often. I am not sure where to look out for. 

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/eade7605-3a70-4398-9ac2-ef9f486c2a5d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Guide to setting up HA

[Lasse Taul Bjerre]

Hi,

I've been playing with the Graylog appliance for a while now.
First it started out as a test, now we are moving towards using it in 
production. For that I need to set it up in HA.
We will be using F5 for load balancing.

As I understand it the OVA is not tuned for real HA, is that correct?

My plan was 3 Graylog server 1 web, 2 identical graylog servers with all 
features except the web interface.
We currently send around 3GB data to the current setup, and I expect it to 
rise to 5GB. We cycle the indice every 24 hours, and keep only the last 40.
So not a big setup, I just would like to have HA, since we are going to use 
this to store/index all our log files.

My Linux skills are minimal but I know the basics.
I have been looking around for a good Graylog HA Howto - but I have not 
found an comprehensive guide. I have found a few guides for setting up a 
single Graylog server.
I have found references to Chef - but my knowledge of Chef is Zero.

Can someone point me in direction of a good guide in acomplising this?

Thank you advance,

Lasse

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/a65ae565-d9c1-4eeb-a4be-bd8453c681cd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.