[graylog2] Ask for feature : trends

[Jean-Luc Bassereau]

Hi all,

Do you plan to add a trending information on dashboards (like kibana can
do) ?

-- 
Cheers,
Jean-Luc Bassereau

-- 
You received this message because you are subscribed to the Google Groups 
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Graylog2 0.91.0-rc.1 "Cluster information currently unavailable"

[Dustin Tennill]

I think have resolved my issue, and I don't think it was related to message 
spooling on disk specifically. I disabled the message_cache_spool_dir 
setting and went back to just using ram. 

Theory: 
The elasticsearch cluster wasn't taking messages fast enough, thus heap 
size continued to grow until GL server crashed or the message spool filled 
up.  

After increasing the number of elasticsearch servers from 2 to 4, GL server 
now runs longer. 
Next, the output_batch_size was changed from 100 to 6000 - the heap size 
now grows less quickly. 

Better. 

Dustin


On Tuesday, October 7, 2014 5:29:34 PM UTC-4, Dustin Tennill wrote:
>
> OK. 
>
> Separate filesystem doesn't resolve the issue, graylog2 just runs until 
> the message_cache_spool_dir fills up and then crashes. 
>
> I am writing a couple of scripts to catch when the disk is nearly full, 
> stop services, delete all the files in the message_cache_spool_dir and 
> start services back up. 
>
> I am going to try a fresh install of graylog on another host and see if 
> the issue occurs. 
>
>
>
>
>
> On Monday, October 6, 2014 10:09:57 AM UTC-4, Dustin Tennill wrote:
>>
>> It crashed once the disk filled up. 
>>
>> I am going create a partition just for the message_cache_spool_dir to see 
>> if perhaps it is aware of full disk and will resolve the issue itself. 
>>
>> Anyone have any specific information on this setting? Documentation 
>> doesn't mention it yet, and I can't see any way to handle it other than 
>> stop/delete files/start. 
>>
>>
>>
>> On Sunday, October 5, 2014 3:22:10 PM UTC-4, Dustin Tennill wrote:
>>>
>>> The spool directory is growing at a steady rate - around 500M every five 
>>> minutes.
>>>
>>> root@myhost:/var/lib/graylog2-server/message-cache-spool# sleep 300; du 
>>> -sh *; date;sleep 300; du -sh *; date;sleep 300; du -sh *; date;
>>> 40Kinput-cache
>>> 664Kinput-cache.p
>>> 900Kinput-cache.t
>>> 61Moutput-cache
>>> 2.9Goutput-cache.p
>>> 904Koutput-cache.t
>>> Sun Oct  5 14:45:50 EDT 2014
>>> 40Kinput-cache
>>> 664Kinput-cache.p
>>> 900Kinput-cache.t
>>> 61Moutput-cache
>>> 3.3Goutput-cache.p
>>> 904Koutput-cache.t
>>> Sun Oct  5 14:50:50 EDT 2014
>>> 40Kinput-cache
>>> 664Kinput-cache.p
>>> 900Kinput-cache.t
>>> 61Moutput-cache
>>> 3.7Goutput-cache.p
>>> 1.7Moutput-cache.t
>>> Sun Oct  5 14:55:50 EDT 2014
>>>
>>> Based on past experience, this will grow until graylog2 crashes. 
>>>
>>>
>>> On Sunday, October 5, 2014 2:18:39 PM UTC-4, Dustin Tennill wrote:

 Apologies to the group - I didn't realize my posts were being moderated 
 until I had attempted post the same comment several times. 

 I enabled the message_cache_off_heap setting and it seems to have 
 resolved slow GC crash issue.
 message_cache_off_heap = true 
 message_cache_spool_dir = /var/lib/graylog2-server/message-cache-spool
 With this setting on, my 20G heap stays between 5G and 10G utilized. 

 However, as far as I can tell the message_cache_spool_dir seems to grow 
 until the disk fills up. 

 Has anyone experienced this? Is there a cleanup operation I should be 
 performing? 

 Dustin


 On Wednesday, October 1, 2014 12:16:19 PM UTC-4, Dustin Tennill wrote:
>
> All,
>
> I recently upgraded to rc.1/ElasticSearch 1.3.2 and am having some 
> issues. We are not in production yet, and I understand that I should 
> expect 
> problems with the release candidate code. 
>
> *Our Graylog Environment:*
> A single Graylog Radio Server (0.91.0-rc.1)
> A single Graylog Server (0.91.0-rc.1)
> Java Settings:  -Xmx20480M -Xms20480M -verbose:gc 
> -Xloggc:/var/log/grayloggc.log -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
> A single Graylog-Web Server (0.91.0-rc.1)
> Two ElasticSearch Nodes (1.3.2)
> Statistics: 6000-7000 msgs per second when things are working correctly
>
> *1. "Cluster information currently unavailable" message shown when I 
> browse to the system page. *
> Since upgrading to the current release, I note that the ElasticSearch 
> health indication page nearly always shows "Cluster information currently 
> unavailable". 
> My ElasticSearch cluster appears healthy to me. I am using the head 
> plugin, and can confirm all is "green" and both nodes are caught up. 
> At least once this has worked correctly - not sure why. 
>
> This doesn't appear to mean anything, data is still coming in and 
> being processed correctly. 
>
> *2. Graylog2-server - crashes eventually due to slow garbage 
> collection. *
> I don't know for sure that this is WHY I seem to have a crash, but the 
> trend seems to be if GC takes longer than a few seconds, I start seeing 
> these message patterns. 
>
> 2014-10-01 11:59:09,598 WARN : org.elasticsearch.monitor.jvm - 
> [graylog2-server] [gc][old]

[graylog2] Correct view of log coming from any Cisco device

[mbaldov]

Hello everybody,

First of all thnaks for exist!
I
''ve just installed and configured a GreyLog2 server with success.
It would be awesome if it could manage correctly logs sent from any Cisco 
devices.
So I have a question for you:
Is it possible to receive in a correct way the logs from Cisco devices with 
a "clean"
(without third party software) installation?

I've tested several solution found on internet like this (that it's seems 
more relevant in my modest opinion):
--



*no service sequence-numbersno service timestamps log datetime msecno 
logging message-counter sysloglogging origin-id hostname*
-

but the result has no changed.

Waitng for a kindly reply.

Best Regards.


-- 
You received this message because you are subscribed to the Google Groups 
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Send data from Graylog2 to Graphite?

[Iain Keddie]

Hi,

This is a bit late in the day, but could you combine some of the logstash 
features with your current pipeline, to get the desired result?  You can 
use logstash to split your output so some goes to graylog via gelf and some 
goes to graphite via the graphite output.  Just a 
thought. http://logstash.net/docs/1.4.2/

 

On Friday, 19 September 2014 22:30:54 UTC+1, Robert Pohl wrote:
>
> Yes that is what I do, but that is showing only the amount of messages, 
> not the aggregated value of the messages (generated by the extractor). Or 
> am I missing something? :)
>
>
> Den tisdagen den 16:e september 2014 kl. 14:41:10 UTC+2 skrev Jochen 
> Schalanda:
>>
>> Am Dienstag, 16. September 2014 13:33:36 UTC+2 schrieb Robert Pohl:
>>>
>>> Is there a way to extract a "value" in the log and present those values 
>>> in a graph on the dashboard?
>>>
>>
>> You could create additional fields in your log messages using some 
>> extractors (see 
>> http://graylog2.org/resources/documentation/general/extractors) and use 
>> those additional fields in your dashboards.
>>
>>
>> Cheers,
>> Jochen
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] sylogd on graylog2 server at port 514 not getting configured in listening mode ?

[Sanchal Singh]

Hi

I configured the below information on the syslogd server as per the 
documentation

Followed the steps:

1. Create an rsyslog configuration file in /etc/rsyslog.d. We will call 
ours 90-graylog2.conf:

sudo vi /etc/rsyslog.d/90-graylog2.conf

2. In this file, add the following lines to configure rsyslog to send 
syslog messages to your Graylog2 server (replace gl2_private_IP with your 
Graylog2 server's private IP address):

#!/bin/bash

$template GRAYLOGRFC5424,"%protocol-version% %timestamp:::date-rfc3339% 
%HOSTNAME% %app-name% %procid% %msg%\n" *.* @:514;GRAYLOGRFC5424

3. Save and quit. This file will be loaded as part of your rsyslog 
configuration from now on. Now you need to restart rsyslog to put your 
change into effect.

sudo service rsyslog restart

On the graylog2-server, the 514 UDP port is configured

The sylogd service is stopped on Graylog2 server




[root@graylog2 ~]# iptables -t nat -L -n -v


Chain PREROUTING (policy ACCEPT 832 packets, 88870 bytes)


pkts bytes target prot opt in out source destination


0 0 REDIRECT udp -- eth0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:514 redir ports 10515


0 0 REDIRECT tcp -- eth0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:514 redir ports 5514


0 0 REDIRECT tcp -- eth0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:514 redir ports 5514


0 0 REDIRECT udp -- eth0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:514 redir ports 2525


[root@graylog2 ~]# netstat -an | grep 514


tcp 0 0 127.0.0.1:27017 127.0.0.1:51498 ESTABLISHED


tcp 0 0 :::127.0.0.1:51498 :::127.0.0.1:27017 ESTABLISHED


udp 0 0 :::10.222.189.50:514 :::*


[root@graylog2 ~]# netstat -an | grep 2525


udp 0 0 :::10.222.189.50:2525 :::*


Note -> they are not running in listening state


Please, help as got stuck at this phase


Thanks and Regards,





-- 
You received this message because you are subscribed to the Google Groups 
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


graylog issue.docx
Description: MS-Word 2007 document

[graylog2] Upgrade from 0.90 to 0.91 breaks "graylog2-web"

[Stanislav Kopp]

I've done upgrade today from 0.90 to 0.91 via debian repos, now I have the 
same error as Sébastien and Jim with graylog2-web in this thread 
https://groups.google.com/forum/#!topic/graylog2/7LU3apz1GmI.

2014-10-21T12:19:44.651+02:00 - [ERROR] - from 
org.graylog2.restclient.lib.ServerNodesRefreshService in 
servernodes-refresh-0 
Resolving configured nodes failed

does anybody came to solution? 

Some info:

log01:/usr/share/graylog2-server/bin# dpkg -l |grep graylog
ii  graylog2-server  0.91.1-1 
 all  Graylog2 server
ii  graylog2-stream-dashboard0.90.0-1 
 all  Graylog2 Stream Dashboard
ii  graylog2-web 0.91.1-1 
 all  Graylog2 web

graylog2-server seems work fine:

2014-10-21T11:44:39.238+02:00 INFO  [RestApiService] Started REST API at 

2014-10-21T11:44:39.238+02:00 INFO  [ServiceManagerListener] Services are 
healthy

log01:/usr/share/graylog2-server/bin# netstat -tulpne |grep 12900
tcp6   0  0 192.168.1.234:12900:::*   
 LISTEN  1001   38350   21742/java

-- 
You received this message because you are subscribed to the Google Groups 
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Getting plain text into Graylog2

[Iain Keddie]

Hi,

Please could you point me in the right direction for a simple way to get 
text from an arbitrary file, into Graylog2?

I have a 40+ text logfiles from a number of web application servers (tomcat 
and weblogic), and I'd like to get the output sent to graylog.  

I can see that the GELF format seems a sensible transport format. Is there 
standard a tool which can effectively "tail" a file and send the data to 
graylog?

I'd also like to send my graphite metrics to graylog as well as graphite, 
to put some alerts in place, so any ideas in that area would be good.

I can see some kind of logstash/greylog hybrid working, but I'd like to 
stick to one platform if I can.

Thanks in advance.

-- 
You received this message because you are subscribed to the Google Groups 
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Server fails to start

[Mark Moorcroft]

I rebooted my graylog2 box today and now I get the following:

[root@graylog ~]# service graylog2-server start
Starting graylog2-server:  [  OK  ]
[root@graylog ~]# Exception in thread "main" java.lang.AssertionError: data 
were read beyond record size, check your serializer

Followed by 2 pages of java errors.

Anybody have any ideas?

-- 
You received this message because you are subscribed to the Google Groups 
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Server fails to start

[Lennart Koopmann]

Hey Mark,

can you post those Java errors/stacktraces?

Thanks,
Lennart

On Thu, Oct 23, 2014 at 12:10 AM, Mark Moorcroft  wrote:
>
> I rebooted my graylog2 box today and now I get the following:
>
> [root@graylog ~]# service graylog2-server start
> Starting graylog2-server:  [  OK  ]
> [root@graylog ~]# Exception in thread "main" java.lang.AssertionError: data
> were read beyond record size, check your serializer
>
> Followed by 2 pages of java errors.
>
> Anybody have any ideas?
>
> --
> You received this message because you are subscribed to the Google Groups
> "graylog2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to graylog2+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Correct view of log coming from any Cisco device

[Lennart Koopmann]

Let your Cisco devices send to a Graylog2 "Raw/Plaintext" input and
use the Graylog2 extractors to parse the message.

On Wed, Oct 22, 2014 at 3:49 PM,   wrote:
> Hello everybody,
>
> First of all thnaks for exist!
> I
> ''ve just installed and configured a GreyLog2 server with success.
> It would be awesome if it could manage correctly logs sent from any Cisco
> devices.
> So I have a question for you:
> Is it possible to receive in a correct way the logs from Cisco devices with
> a "clean"
> (without third party software) installation?
>
> I've tested several solution found on internet like this (that it's seems
> more relevant in my modest opinion):
> --
> no service sequence-numbers
> no service timestamps log datetime msec
> no logging message-counter syslog
> logging origin-id hostname
> -
>
> but the result has no changed.
>
> Waitng for a kindly reply.
>
> Best Regards.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "graylog2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to graylog2+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Server fails to start

[Mark Moorcroft]

Thanks, I reverted my VM image and solved it that way.

On Wednesday, October 22, 2014 3:58:50 PM UTC-7, lennart wrote:
>
> Hey Mark, 
>
> can you post those Java errors/stacktraces? 
>
> Thanks, 
> Lennart 
>
> On Thu, Oct 23, 2014 at 12:10 AM, Mark Moorcroft  > wrote: 
> > 
> > I rebooted my graylog2 box today and now I get the following: 
> > 
> > [root@graylog ~]# service graylog2-server start 
> > Starting graylog2-server:  [  OK  ] 
> > [root@graylog ~]# Exception in thread "main" java.lang.AssertionError: 
> data 
> > were read beyond record size, check your serializer 
> > 
> > Followed by 2 pages of java errors. 
> > 
> > Anybody have any ideas? 
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups 
> > "graylog2" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> an 
> > email to graylog2+u...@googlegroups.com . 
> > For more options, visit https://groups.google.com/d/optout. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.