date:20160627

[graylog2] Colors in Charts and Data Table

2016-06-27 Thread 'Joshua Humpich' via Graylog Users

Hello,
is there a way to manipulate the colors of a pie charts or data table views?
At the moment my application log levels (info, debug, trace, error) got bad 
colors for the chart.
The idea is to tell graylog which log level wich color gets or something.

Regards,
Josh

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/8743580b-90fb-4152-b0a9-22da71dd801e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Adding Custom Functions for defining Rules

2016-06-27 Thread Mayur Mangalampalli

Hi,

In the documentation for graylog. It is written that Functions are written 
in Java and are pluggable, allowing extending the capabilities of Graylog. 
There is no place in documentation where I could find how I could add 
custom Functions. I also looked at the source code from the following 
https://github.com/Graylog2/graylog-plugin-pipeline-processor

I followed the instructions on how to create a plugin and tried to add my 
function by copying certain classes from the above package and building a 
custom plugin. Here are the steps that I followed to do that.

1) Created a plugin skeleton project based on the instructions provided in 
the Plugins section
2) Created a function called to_integer. Copied all the required classes to 
build the function. Here is my sample code to create the to_integer 
function.

public class IntegerConversion extends AbstractFunction {


public static final String NAME = "to_integer";


private static final String VALUE = "value";

private static final String DEFAULT = "default";

private final ParameterDescriptor valueParam;

private final ParameterDescriptor defaultParam;


public IntegerConversion() {

valueParam = object(VALUE).build();

defaultParam = toint(DEFAULT).optional().build();

}


@Override

public Integer evaluate(FunctionArgs args, EvaluationContext context) {

final Object evaluated = valueParam.required(args, context);

final Integer defaultValue = defaultParam.optional(args, context
).orElse(0);

if (evaluated == null) {

return defaultValue;

}

return firstNonNull(Ints.tryParse(evaluated.toString()),

defaultValue);

}


@Override

public FunctionDescriptor descriptor() {

return FunctionDescriptor.builder()

.name(NAME)

.returnType(Integer.class)

.params(of(

valueParam,

defaultParam

))

.build();

}

}

3) I was not sure what should be done after that. I also added the Function 
the FunctionPluginModule and then installed it. But I did not see any error 
or the function getting created. Please let me know if I am missing 
something.

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/3d6971cc-37d5-41d1-b688-4945b6a0d193%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Kafka output plugin for Graylog 2.x?

2016-06-27 Thread Frederic Desjarlais


Is anyone aware of a Kafka output plugin for Graylog 2.x?  If not, is 
Graylog itself considering creating/maintaining such a plugin in the near 
future?

We're considering building one ourselves, but we'd like to ensure one 
doesn't already exist (or in the works).  We didn't find such a plugin in 
the Marketplace (https://marketplace.graylog.org/addons?search=kafka), nor 
via a regular Google query.

Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/9390bcfb-40d1-4c8d-8b16-28e70307a568%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Having some difficulties with 3 node graylog cluster

2016-06-27 Thread Yiannis

Hi Jan,
I've send you the log file by mail.

We're using LDAP authentication (Actually active directory)
but we do not face any login problems.
Everything works as expected except the search button mentioned in the 
previous mails.

Thank you for your time

Regards
Yiannis


On Monday, June 27, 2016 at 1:28:36 PM UTC+2, Jan Doberstein wrote:
>
> Hej Yiannis, 
>
>
>
>
> On 24. Juni 2016 at 16:19:01, Yiannis (ka...@stoiximan.gr ) 
> wrote: 
> > the setup is really straight forward and never thought that i will have 
> > difficulties but…. 
>
> you are faced a strange issue. That looks like a corner-case. 
>
> > 
> > On Friday, June 24, 2016 at 10:42:21 AM UTC+2, Jan Doberstein wrote: 
>
> > That is my starting papameters for all graylog server 
> > GRAYLOG_SERVER_JAVA_OPTS= 
> > "-Xms8g -Xmx8g -XX:NewRatio=1 -server -XX:+ResizeTLAB 
> > -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled 
> > -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC 
> > -XX:-OmitStackTraceInFastThrow" 
> > 
>
> > > > My 2 biggest problem are: 
> > > > 
> > > > 1) Most of the times when i press the search button (and only the 
> search 
> > > > button displayed in the image) 
> > > > 
> > > > seems to me that my browser goes again from the login screen (to 
> send 
> > > again 
> > > > the user credential) before rendering the results 
> > > 
> > > Can you please look into your log files of graylog when this happens 
> > > to you - it should be possible to get an idea why this happen just by 
> > > look at the log file during this ‘event’. 
>
> > When the log lever is INFO nothing appears in the log during this 
> ‘event’, 
> > when i change to DEBUG or TRACE i really can't get the idea of what is 
> > happening. 
>
> if possible can you upload such logs somewhere - just because this is 
> something Graylog related and not Elasticsearch or Mongo. 
> This would be really helpful to help. 
>
>
>
> > > > 2) Every now and then, i get a strange error (when mostly when using 
> > > > firefox) from webs interface api server like the following 
> > > > (no errors on shown in the graylog server logs) 
> > > 
> > > Are you sure that you read 
> > > 
> > > 
> http://docs.graylog.org/en/2.0/pages/configuration/web_interface.html#overview
>  
> > > 
> > > and set all Configurations to that? 
> > > 
> > > Even if you run the Web Interface only on one Node the API of all 
> > > Nodes need to be reachable by your browser. 
> > > 
> > > 
> > I believe i did 
>
> > and yes the API of all Nodes is reachable from my browser. 
>
> It could be that you are faced the issue Jason mentioned in his Mail - 
> maybe you can give us some Information that we are able to reproduce. 
>
> thx 
> Jan 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/5c2837ff-7f67-4e95-91f3-0c8e22e10fa0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Alerting for flopping ports

2016-06-27 Thread Emil Grama


I'm new with graylog and maybe one of you guys can help:
I have in graylog lots of  events of the type  application 
online/application offline generated by thousands of different clients.
I would like to create an alert that is triggered if I gent an offline from 
hostA and in 30 minutes I do not get an online from hostA.
Is this possible to do with graylog? If so please point me to the right 
direction; the alert rules I found in the documentation were pretty basic. 
Maybe some plugin?

Thank you!

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/d998d0d1-f08d-4bf3-9bac-7471828d593f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Backup of indices in Graylog 1.3

2016-06-27 Thread robertocarna36

Hi people, I have Graylog 1.3 as my syslog server. I have setup the 
following strategy:

10 indices
3 days for indice
delete and not close 
total: 30 days of data

I want to backup the indices to a Networker EMC server, but all the indices 
I have in the Graylog web interface are not closed.

Can I backup a non-closed indice (or index) ??? Or when I have to restore 
it after a long time I will can't do that because the indice was not closed 
???

Thanks a lot, regards.

Roberto

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/cd30e7a4-3881-4e38-9011-c1c07079a710%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Need some help disabling ciphers and algorithms

2016-06-27 Thread Ragnar

Ah great, I'll give that a try and report back, thanks a lot.

On Monday, June 27, 2016 at 5:16:25 PM UTC+3, Marius Sturm wrote:
>
> Ah ok, than you can use the advanced attributes in 
> /etc/graylog/graylog-settings.json and modify these values: 
> https://github.com/Graylog2/omnibus-graylog2/blob/2.0/files/graylog-cookbooks/graylog/attributes/default.rb#L47-L48
>
> On 27 June 2016 at 16:06, Ragnar  
> wrote:
>
>> Hi Marius,
>>
>> Just for the web interface, our security department flagged the system as 
>> being vulnerable to heartbleed/POODLE/FROWN etc. because SSLv2 and SSLv3 
>> are enabled (along with weak RC4 ciphers). 
>>
>> On Monday, June 27, 2016 at 3:59:56 PM UTC+3, Marius Sturm wrote:
>>>
>>> @Ragnar do you try to disable the cipher algorithms for the web 
>>> interface or for an log input? Because the web interface on the appliances 
>>> is TLS terminated by the Nginx that is also installed. The inputs are 
>>> served directly by Graylog's java process, that whould be a differnet 
>>> setting.
>>>
>>> On 27 June 2016 at 13:46, Jan Doberstein  wrote:
>>>
 Hej Ragnar,



 On 25. Juni 2016 at 14:13:32, Ragnar (invalid...@gmail.com) wrote:
 > Steps Tried:
 > 1. Created a security.properties file using the exact example
 > (un-commenting out the relevant lines) and put it in the
 > /opt/graylog/server directory
 > 2. Ran the command java
 > -Djava.security.properties=/opt/graylog/server/security.properties 
 -jar
 > /opt/graylog/server/graylog.jar server
 >
 > Received an error staying that etc/graylog/server/server.conf didn't 
 exist
 > so I created it
 >
 > 3. Ran the command java
 > -Djava.security.properties=/opt/graylog/server/security.properties 
 -jar
 > /opt/graylog/server/graylog.jar server again and now I get the error:

 > Any ideas?

 you need to add as additional startup parameter to graylog!

 as you use graylog OVA image i had created this issue:
 https://github.com/Graylog2/omnibus-graylog2/issues/31

 because this is not save possible.

 /jd

 --
 You received this message because you are subscribed to the Google 
 Groups "Graylog Users" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to graylog2+u...@googlegroups.com.
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/graylog2/CAGm-bLb4v0JHLz5acB2A6s6dYqH31fNUU_Y3OM8PVijFYhCD3w%40mail.gmail.com
 .
 For more options, visit https://groups.google.com/d/optout.

>>>
>>>
>>>
>>> -- 
>>> Developer
>>>
>>> Tel.: +49 (0)40 609 452 077
>>> Fax.: +49 (0)40 609 452 078
>>>
>>> TORCH GmbH - A Graylog Company
>>> Poolstraße 21
>>> 20335 Hamburg
>>> Germany
>>>
>>> https://www.graylog.com 
>>>
>>> Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175
>>> Geschäftsführer: Lennart Koopmann (CEO)
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Graylog Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to graylog2+u...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/graylog2/2bf726e3-095e-4a13-a5a6-da07c70783c9%40googlegroups.com
>>  
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> -- 
> Developer
>
> Tel.: +49 (0)40 609 452 077
> Fax.: +49 (0)40 609 452 078
>
> TORCH GmbH - A Graylog Company
> Poolstraße 21
> 20335 Hamburg
> Germany
>
> https://www.graylog.com 
>
> Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175
> Geschäftsführer: Lennart Koopmann (CEO)
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/d785f608-61be-4768-843e-f67c112b8c3a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Need some help disabling ciphers and algorithms

2016-06-27 Thread Ragnar

Hi Marius,

Just for the web interface, our security department flagged the system as 
being vulnerable to heartbleed/POODLE/FROWN etc. because SSLv2 and SSLv3 
are enabled (along with weak RC4 ciphers). 

On Monday, June 27, 2016 at 3:59:56 PM UTC+3, Marius Sturm wrote:
>
> @Ragnar do you try to disable the cipher algorithms for the web interface 
> or for an log input? Because the web interface on the appliances is TLS 
> terminated by the Nginx that is also installed. The inputs are served 
> directly by Graylog's java process, that whould be a differnet setting.
>
> On 27 June 2016 at 13:46, Jan Doberstein  
> wrote:
>
>> Hej Ragnar,
>>
>>
>>
>> On 25. Juni 2016 at 14:13:32, Ragnar (invalid...@gmail.com ) 
>> wrote:
>> > Steps Tried:
>> > 1. Created a security.properties file using the exact example
>> > (un-commenting out the relevant lines) and put it in the
>> > /opt/graylog/server directory
>> > 2. Ran the command java
>> > -Djava.security.properties=/opt/graylog/server/security.properties -jar
>> > /opt/graylog/server/graylog.jar server
>> >
>> > Received an error staying that etc/graylog/server/server.conf didn't 
>> exist
>> > so I created it
>> >
>> > 3. Ran the command java
>> > -Djava.security.properties=/opt/graylog/server/security.properties -jar
>> > /opt/graylog/server/graylog.jar server again and now I get the error:
>>
>> > Any ideas?
>>
>> you need to add as additional startup parameter to graylog!
>>
>> as you use graylog OVA image i had created this issue:
>> https://github.com/Graylog2/omnibus-graylog2/issues/31
>>
>> because this is not save possible.
>>
>> /jd
>>
>> --
>> You received this message because you are subscribed to the Google Groups 
>> "Graylog Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to graylog2+u...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/graylog2/CAGm-bLb4v0JHLz5acB2A6s6dYqH31fNUU_Y3OM8PVijFYhCD3w%40mail.gmail.com
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> -- 
> Developer
>
> Tel.: +49 (0)40 609 452 077
> Fax.: +49 (0)40 609 452 078
>
> TORCH GmbH - A Graylog Company
> Poolstraße 21
> 20335 Hamburg
> Germany
>
> https://www.graylog.com 
>
> Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175
> Geschäftsführer: Lennart Koopmann (CEO)
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/2bf726e3-095e-4a13-a5a6-da07c70783c9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Need some help disabling ciphers and algorithms

2016-06-27 Thread Ragnar

Hey Jan,

Thanks for your reply, so if I understand correctly this is only possible 
(currently) for non-OVA implementations, is that correct?


On Monday, June 27, 2016 at 2:46:29 PM UTC+3, Jan Doberstein wrote:
>
> Hej Ragnar, 
>
>
>
> On 25. Juni 2016 at 14:13:32, Ragnar (invalid...@gmail.com ) 
> wrote: 
> > Steps Tried: 
> > 1. Created a security.properties file using the exact example 
> > (un-commenting out the relevant lines) and put it in the 
> > /opt/graylog/server directory 
> > 2. Ran the command java 
> > -Djava.security.properties=/opt/graylog/server/security.properties -jar 
> > /opt/graylog/server/graylog.jar server 
> > 
> > Received an error staying that etc/graylog/server/server.conf didn't 
> exist 
> > so I created it 
> > 
> > 3. Ran the command java 
> > -Djava.security.properties=/opt/graylog/server/security.properties -jar 
> > /opt/graylog/server/graylog.jar server again and now I get the error: 
>
> > Any ideas? 
>
> you need to add as additional startup parameter to graylog! 
>
> as you use graylog OVA image i had created this issue: 
> https://github.com/Graylog2/omnibus-graylog2/issues/31 
>
> because this is not save possible. 
>
> /jd 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/8cb40ba9-b403-49a2-971f-a04fe5752e68%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Re: Anyone use Image in real world application? Graylog 2.0 image fails after few days. Is this Image problem or Graylog in general?

2016-06-27 Thread John

screenshots from my UI






בתאריך יום שני, 27 ביוני 2016 בשעה 15:32:52 UTC+3, מאת John:
>
> Hi
> I checked the elasticsearch log and I don't see something special
> The cluster status is green
>
> this is the last log file
>
> 2016-06-26_09:51:28.78352 [2016-06-26 12:51:28,782][INFO ][node   
>   ] [Glenn Talbot] version[2.3.1], pid[953], 
> build[bd98092/2016-04-04T12:25:05Z]
> 2016-06-26_09:51:28.79783 [2016-06-26 12:51:28,794][INFO ][node   
>   ] [Glenn Talbot] initializing ...
> 2016-06-26_09:51:30.17146 [2016-06-26 12:51:30,171][INFO ][plugins 
>  ] [Glenn Talbot] modules [reindex, lang-expression, lang-groovy], 
> plugins [kopf], sites [kopf]
> 2016-06-26_09:51:30.29289 [2016-06-26 12:51:30,292][INFO ][env 
>  ] [Glenn Talbot] using [1] data paths, mounts [[/ 
> (/dev/mapper/graylog--vg-root)]], net usable_space [11gb], net total_space 
> [14.9gb], spins? [possibly], types [ext4]
> 2016-06-26_09:51:30.29564 [2016-06-26 12:51:30,294][INFO ][env 
>  ] [Glenn Talbot] heap size [37.6gb], compressed ordinary object 
> pointers [false]
> 2016-06-26_09:51:30.29766 [2016-06-26 12:51:30,294][WARN ][env 
>  ] [Glenn Talbot] max file descriptors [64000] for elasticsearch 
> process likely too low, consider increasing to at least [65536]
> 2016-06-26_09:51:34.69050 [2016-06-26 12:51:34,690][INFO ][node   
>   ] [Glenn Talbot] initialized
> 2016-06-26_09:51:34.69107 [2016-06-26 12:51:34,690][INFO ][node   
>   ] [Glenn Talbot] starting ...
> 2016-06-26_09:51:35.32863 [2016-06-26 12:51:35,328][INFO ][transport   
>  ] [Glenn Talbot] publish_address {172.25.232.45:9300}, 
> bound_addresses {172.25.232.45:9300}
> 2016-06-26_09:51:35.33658 [2016-06-26 12:51:35,336][INFO ][discovery   
>  ] [Glenn Talbot] graylog-production/th7wM-a9ThaAY_umCV3v2w
> 2016-06-26_09:51:45.37933 [2016-06-26 12:51:45,379][INFO ][cluster.service 
>  ] [Glenn Talbot] new_master {Glenn 
> Talbot}{th7wM-a9ThaAY_umCV3v2w}{172.25.232.45}{172.25.232.45:9300}, added 
> {{graylog-a0b12869-11ed-4d89-ae58-dcc7380bc3b8}{KA4cjlTpQTm9Y1Rv5wlVmw}{172.25.232.41}{172.25.232.41:9350}{client=true,
>  
> data=false, 
> master=false},{graylog-2a34-d1ba-4f21-a9df-f45901d845b7}{BiWe2Zy2Syaojr9ek0AlJQ}{172.25.232.35}{172.25.232.35:9350}{client=true,
>  
> data=false, master=false},}, reason: zen-disco-join(elected_as_master, [0] 
> joins received)
> 2016-06-26_09:51:45.40239 [2016-06-26 12:51:45,402][INFO ][http   
>   ] [Glenn Talbot] publish_address {172.25.232.45:9200}, 
> bound_addresses {172.25.232.45:9200}
> 2016-06-26_09:51:45.40350 [2016-06-26 12:51:45,403][INFO ][node   
>   ] [Glenn Talbot] started
> 2016-06-26_09:51:45.53808 [2016-06-26 12:51:45,537][INFO ][gateway 
>  ] [Glenn Talbot] recovered [1] indices into cluster_state
> 2016-06-26_09:51:45.87525 [2016-06-26 12:51:45,875][INFO 
> ][cluster.routing.allocation] [Glenn Talbot] Cluster health status changed 
> from [RED] to [GREEN] (reason: [shards started [[graylog_0][0]] ...]).
> 2016-06-26_09:57:01.91281 [2016-06-26 12:57:01,912][INFO ][cluster.service 
>  ] [Glenn Talbot] added 
> {{graylog-c1be9fdd-8c8a-41b1-8a2f-dacbddbc0cc5}{-7icx5UPSrWbs9jqXVE2Mg}{172.25.232.36}{172.25.232.36:9350}{client=true,
>  
> data=false, master=false},}, reason: zen-disco-join(join from 
> node[{graylog-c1be9fdd-8c8a-41b1-8a2f-dacbddbc0cc5}{-7icx5UPSrWbs9jqXVE2Mg}{172.25.232.36}{172.25.232.36:9350}{client=true,
>  
> data=false, master=false}])
> 2016-06-26_10:17:02.43148 [2016-06-26 13:17:02,428][INFO 
> ][cluster.metadata ] [Glenn Talbot] [graylog_0] update_mapping 
> [message]
> 2016-06-26_15:35:13.25159 [2016-06-26 18:35:13,250][INFO ][node   
>   ] [Glenn Talbot] stopping ...
> 2016-06-26_15:35:13.32027 [2016-06-26 18:35:13,319][INFO ][node   
>   ] [Glenn Talbot] stopped
> 2016-06-26_15:35:13.32153 [2016-06-26 18:35:13,320][INFO ][node   
>   ] [Glenn Talbot] closing ...
> 2016-06-26_15:35:13.33032 [2016-06-26 18:35:13,329][INFO ][node   
>   ] [Glenn Talbot] closed
> 2016-06-26_15:46:49.97957 [2016-06-26 18:46:49,977][INFO ][node   
>   ] [Tether] version[2.3.1], pid[1364], 
> build[bd98092/2016-04-04T12:25:05Z]
> 2016-06-26_15:46:49.97959 [2016-06-26 18:46:49,978][INFO ][node   
>   ] [Tether] initializing ...
> 2016-06-26_15:46:50.52052 [2016-06-26 18:46:50,519][INFO ][plugins 
>  ] [Tether] modules [reindex, lang-expression, lang-groovy], 
> plugins [kopf], sites [kopf]
>

[graylog2] Syslog messages look different between Splunk and Graylog

2016-06-27 Thread Keamas M

Hello,
I am new to graylog. I used Splunk before but I reached the space limit of 
splunk. Thats why I installed Graylog.
I want to log firewall Logs and create reports and graphs out of this Logs.

   - how similar is the Search syntax between Splunk and Graylog? Is it 
   complicated to migrate this?



   - But the main issue at the moment is that the syslog messages which I 
   get are different if you compare graylog and Splunk
   

Splunk Syslog message:

<14>Jun 27 12:27:30 FW-02 2/C1/WN02/box_Firewall_Activity: Info C-WN02-FW 
Detect: type=FWD|proto=TCP|srcIF=port7.101|srcIP=10.244.130.143|srcPort=
52365|srcMAC=00:00:00:00:00:00|dstIP=194.232.104.167|dstPort=80|dstService=|
dstIF=port7.910|rule=|info=Normal Operation|srcNAT=80.120.132.156|dstNAT=194
.232.154.127|duration=0|count=1|receivedBytes=0|sentBytes=0|receivedPackets=
0|sentPackets=0|user=n600771|protocol=HTTP direct|application=Web browsing|
target=steiermark.orf.at|content=|urlcat=Search Engines/Portals


Graylog Syslog message:

message
NG_Firewall[]: 1467031812 1 10.244.120.142 194.232.112.146 image/png 
10.244.120.142 
http://steiermark.orf.at/mojo/1_3/storyserver/oeka/images/arrow.right.png 
1020 BYF ALLOWED CLEAN 2 1 0 0 0 (-) 0 Search-Engines/Portals 0 - 0 
steiermark.orf.at 
Search-Engines/Portals [00user] steiermark.orf.at - - 0

How can I receive or display the Syslogs in the same format like in Splunk.I 
installed on my Splunk installation this App: 
https://splunkbase.splunk.com/app/2634/
The Syslog Logs have mor informations like SrcNAT, dstNAT and so on. Also a 
name like target= or urlcat=How can I change this settings ? On Splunk 
there was no additional configuration needed.







-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/50f12601-9526-48d5-8641-aac72e8c86c4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Need some help disabling ciphers and algorithms

2016-06-27 Thread Marius Sturm

@Ragnar do you try to disable the cipher algorithms for the web interface
or for an log input? Because the web interface on the appliances is TLS
terminated by the Nginx that is also installed. The inputs are served
directly by Graylog's java process, that whould be a differnet setting.

On 27 June 2016 at 13:46, Jan Doberstein  wrote:

> Hej Ragnar,
>
>
>
> On 25. Juni 2016 at 14:13:32, Ragnar (invalid.nore...@gmail.com) wrote:
> > Steps Tried:
> > 1. Created a security.properties file using the exact example
> > (un-commenting out the relevant lines) and put it in the
> > /opt/graylog/server directory
> > 2. Ran the command java
> > -Djava.security.properties=/opt/graylog/server/security.properties -jar
> > /opt/graylog/server/graylog.jar server
> >
> > Received an error staying that etc/graylog/server/server.conf didn't
> exist
> > so I created it
> >
> > 3. Ran the command java
> > -Djava.security.properties=/opt/graylog/server/security.properties -jar
> > /opt/graylog/server/graylog.jar server again and now I get the error:
>
> > Any ideas?
>
> you need to add as additional startup parameter to graylog!
>
> as you use graylog OVA image i had created this issue:
> https://github.com/Graylog2/omnibus-graylog2/issues/31
>
> because this is not save possible.
>
> /jd
>
> --
> You received this message because you are subscribed to the Google Groups
> "Graylog Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to graylog2+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/graylog2/CAGm-bLb4v0JHLz5acB2A6s6dYqH31fNUU_Y3OM8PVijFYhCD3w%40mail.gmail.com
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Developer

Tel.: +49 (0)40 609 452 077
Fax.: +49 (0)40 609 452 078

TORCH GmbH - A Graylog Company
Poolstraße 21
20335 Hamburg
Germany

https://www.graylog.com 

Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175
Geschäftsführer: Lennart Koopmann (CEO)

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/CAMqbBb%2BKxoAYzGJB-mXdM0jkG%3Dn2aopiRq1ESeD5VrT__eqd1A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Re: Anyone use Image in real world application? Graylog 2.0 image fails after few days. Is this Image problem or Graylog in general?

2016-06-27 Thread John

Hi
I checked the elasticsearch log and I don't see something special
The cluster status is green

this is the last log file

2016-06-26_09:51:28.78352 [2016-06-26 12:51:28,782][INFO ][node 
] [Glenn Talbot] version[2.3.1], pid[953], 
build[bd98092/2016-04-04T12:25:05Z]
2016-06-26_09:51:28.79783 [2016-06-26 12:51:28,794][INFO ][node 
] [Glenn Talbot] initializing ...
2016-06-26_09:51:30.17146 [2016-06-26 12:51:30,171][INFO ][plugins 
 ] [Glenn Talbot] modules [reindex, lang-expression, lang-groovy], 
plugins [kopf], sites [kopf]
2016-06-26_09:51:30.29289 [2016-06-26 12:51:30,292][INFO ][env 
 ] [Glenn Talbot] using [1] data paths, mounts [[/ 
(/dev/mapper/graylog--vg-root)]], net usable_space [11gb], net total_space 
[14.9gb], spins? [possibly], types [ext4]
2016-06-26_09:51:30.29564 [2016-06-26 12:51:30,294][INFO ][env 
 ] [Glenn Talbot] heap size [37.6gb], compressed ordinary object 
pointers [false]
2016-06-26_09:51:30.29766 [2016-06-26 12:51:30,294][WARN ][env 
 ] [Glenn Talbot] max file descriptors [64000] for elasticsearch 
process likely too low, consider increasing to at least [65536]
2016-06-26_09:51:34.69050 [2016-06-26 12:51:34,690][INFO ][node 
] [Glenn Talbot] initialized
2016-06-26_09:51:34.69107 [2016-06-26 12:51:34,690][INFO ][node 
] [Glenn Talbot] starting ...
2016-06-26_09:51:35.32863 [2016-06-26 12:51:35,328][INFO ][transport   
 ] [Glenn Talbot] publish_address {172.25.232.45:9300}, 
bound_addresses {172.25.232.45:9300}
2016-06-26_09:51:35.33658 [2016-06-26 12:51:35,336][INFO ][discovery   
 ] [Glenn Talbot] graylog-production/th7wM-a9ThaAY_umCV3v2w
2016-06-26_09:51:45.37933 [2016-06-26 12:51:45,379][INFO ][cluster.service 
 ] [Glenn Talbot] new_master {Glenn 
Talbot}{th7wM-a9ThaAY_umCV3v2w}{172.25.232.45}{172.25.232.45:9300}, added 
{{graylog-a0b12869-11ed-4d89-ae58-dcc7380bc3b8}{KA4cjlTpQTm9Y1Rv5wlVmw}{172.25.232.41}{172.25.232.41:9350}{client=true,
 
data=false, 
master=false},{graylog-2a34-d1ba-4f21-a9df-f45901d845b7}{BiWe2Zy2Syaojr9ek0AlJQ}{172.25.232.35}{172.25.232.35:9350}{client=true,
 
data=false, master=false},}, reason: zen-disco-join(elected_as_master, [0] 
joins received)
2016-06-26_09:51:45.40239 [2016-06-26 12:51:45,402][INFO ][http 
] [Glenn Talbot] publish_address {172.25.232.45:9200}, 
bound_addresses {172.25.232.45:9200}
2016-06-26_09:51:45.40350 [2016-06-26 12:51:45,403][INFO ][node 
] [Glenn Talbot] started
2016-06-26_09:51:45.53808 [2016-06-26 12:51:45,537][INFO ][gateway 
 ] [Glenn Talbot] recovered [1] indices into cluster_state
2016-06-26_09:51:45.87525 [2016-06-26 12:51:45,875][INFO 
][cluster.routing.allocation] [Glenn Talbot] Cluster health status changed 
from [RED] to [GREEN] (reason: [shards started [[graylog_0][0]] ...]).
2016-06-26_09:57:01.91281 [2016-06-26 12:57:01,912][INFO ][cluster.service 
 ] [Glenn Talbot] added 
{{graylog-c1be9fdd-8c8a-41b1-8a2f-dacbddbc0cc5}{-7icx5UPSrWbs9jqXVE2Mg}{172.25.232.36}{172.25.232.36:9350}{client=true,
 
data=false, master=false},}, reason: zen-disco-join(join from 
node[{graylog-c1be9fdd-8c8a-41b1-8a2f-dacbddbc0cc5}{-7icx5UPSrWbs9jqXVE2Mg}{172.25.232.36}{172.25.232.36:9350}{client=true,
 
data=false, master=false}])
2016-06-26_10:17:02.43148 [2016-06-26 13:17:02,428][INFO ][cluster.metadata 
] [Glenn Talbot] [graylog_0] update_mapping [message]
2016-06-26_15:35:13.25159 [2016-06-26 18:35:13,250][INFO ][node 
] [Glenn Talbot] stopping ...
2016-06-26_15:35:13.32027 [2016-06-26 18:35:13,319][INFO ][node 
] [Glenn Talbot] stopped
2016-06-26_15:35:13.32153 [2016-06-26 18:35:13,320][INFO ][node 
] [Glenn Talbot] closing ...
2016-06-26_15:35:13.33032 [2016-06-26 18:35:13,329][INFO ][node 
] [Glenn Talbot] closed
2016-06-26_15:46:49.97957 [2016-06-26 18:46:49,977][INFO ][node 
] [Tether] version[2.3.1], pid[1364], 
build[bd98092/2016-04-04T12:25:05Z]
2016-06-26_15:46:49.97959 [2016-06-26 18:46:49,978][INFO ][node 
] [Tether] initializing ...
2016-06-26_15:46:50.52052 [2016-06-26 18:46:50,519][INFO ][plugins 
 ] [Tether] modules [reindex, lang-expression, lang-groovy], 
plugins [kopf], sites [kopf]
2016-06-26_15:46:50.54693 [2016-06-26 18:46:50,546][INFO ][env 
 ] [Tether] using [1] data paths, mounts [[/ 
(/dev/mapper/graylog--vg-root)]], net usable_space [11gb], net total_space 
[14.9gb], spins? [possibly], types [ext4]
2016-06-26_15:46:50.54734 [2016-06-26 18:46:50,546][INFO ][env 
 ] [Tether] heap size [37.6gb], compressed ordinary object pointers 
[false]
2016-06-26_15:46:50.54871 [2016-06-26 18:46:50,547][WARN ][env 
 ] [Tether] max file descriptors [64000] for

Re: [graylog2] Re: Anyone use Image in real world application? Graylog 2.0 image fails after few days. Is this Image problem or Graylog in general?

2016-06-27 Thread Marius Sturm

Hi,
this all boils down to an unstable Elasticsearch instance. When Graylog is
not able to forward log messages to ES it buffers them on disk and tries to
send them later. This is called journal.
So when your ES service is not running properly the journal fills up with
messages. Please take a look into the ES logs to figure out why it has
problems with message ingestion. You can find them in
/var/log/graylog/elasticsearch/current

Cheers,
Marius

On 27 June 2016 at 13:39, John  wrote:

> 1 and 4
> and the graylog server node is not sending data to elasticsearch
> I deleted the journal but it doesn't help
> the problems began few days after I upgraded from 1.3 to 2.0.2
>
> בתאריך יום שני, 27 ביוני 2016 בשעה 14:30:28 UTC+3, מאת Joe K:
>
>> Which problem out of 4?
>>
>>
>> On Monday, June 27, 2016 at 2:00:14 PM UTC+3, John wrote:
>>>
>>> Hi Joe
>>> I have exactly the same problem few days after I upgraded from 1.3 to
>>> 2.0.2
>>> Did you managed to fix this issue?
>>>
>>> בתאריך יום חמישי, 26 במאי 2016 בשעה 14:02:19 UTC+3, מאת Joe K:

 - We run it on t2.medium. (4GB RAM, 2 cores)
 - About 1 incoming message per second.
 - tried 2.0.0 and now running 2.0.1

 Anyone use Image in real world application? Graylog 2.0 image fails
 after few days. Is this Image problem or Graylog in general?

 It runs fine for about a week. After that there's errors and search
 stop working. Search requests timeout.
 There's many errors and they are very cryptic, google search does not
 give any solutions how to manage them:

 *1. After about a week we have error "Uncommited messages deleted from
 journal"*

> Uncommited messages deleted from journal (triggered 9 days ago)
> Some messages were deleted from the Graylog journal before they could
> be written to Elasticsearch. Please verify that your Elasticsearch cluster
> is healthy and fast enough. You may also want to review your Graylog
> journal settings and set a higher limit. (Node: f12...

 What to do about this? What is "journal"? Google search produce no
 answers.

 *2. After about 4 days of clean install it always trigger "Cluster
 unhealthy"*

>  "Elasticsearch cluster unhealthy (RED)"
> "The Elasticsearch cluster state is RED which means shards are
> unassigned. This usually indicates a crashed and corrupt cluster and needs
> to be investigated. Graylog will write into the local disk journal. Read
> how to fix this in the Elasticsearch setup documentation."

 When you go to that documentation link it says "The red status
 indicates that some or all of the primary shards are not available. In this
 state, no searches can be performed until all primary shards are restored."
 That's it. what are you supposed to do?
 After long search finally found one solution: this was cured once with 
 *curl
 -XPUT 'localhost:9200/_settings' -d '{ "index" : {
  "number_of_replicas" : 0}}'*
 Next time it happened, we tried the solution again, but response was
 *{"acknowledged":false}*
 So what now???

 *3. Every time we perform graylog-ctl restart four more unassigled
 shards appear:*
  Elasticsearch cluster is yellow. Shards: 20 active, 0 initializing, 0
  relocating, 8 unassigned
 graylog-ctl restart
  Elasticsearch cluster is yellow. Shards: 20 active, 0 initializing, 0
  relocating, 12 unassigned
 Etc.

 *4. Journal utilization is too high without any hint on how to set it
 to higher.*

>  Journal utilization is too high (triggered 11 days ago)
> Journal utilization is too high and may go over the limit soon. Please
> verify that your Elasticsearch cluster is healthy and fast enough. You may
> also want to review your Graylog journal settings and set a higher limit.
> (Node: f121

 What is this "journal"? and how to set it to "higher"?

 Please help!

 --
> You received this message because you are subscribed to the Google Groups
> "Graylog Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to graylog2+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/graylog2/2288cbf2-6f37-4e77-8c32-c50ba64fe71e%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
Developer

Tel.: +49 (0)40 609 452 077
Fax.: +49 (0)40 609 452 078

TORCH GmbH - A Graylog Company
Poolstraße 21
20335 Hamburg
Germany

https://www.graylog.com 

Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175
Geschäftsführer: Lennart Koopmann (CEO)

-- 
You received this message because

Re: [graylog2] Need some help disabling ciphers and algorithms

2016-06-27 Thread Jan Doberstein

Hej Ragnar,



On 25. Juni 2016 at 14:13:32, Ragnar (invalid.nore...@gmail.com) wrote:
> Steps Tried:
> 1. Created a security.properties file using the exact example
> (un-commenting out the relevant lines) and put it in the
> /opt/graylog/server directory
> 2. Ran the command java
> -Djava.security.properties=/opt/graylog/server/security.properties -jar
> /opt/graylog/server/graylog.jar server
>
> Received an error staying that etc/graylog/server/server.conf didn't exist
> so I created it
>
> 3. Ran the command java
> -Djava.security.properties=/opt/graylog/server/security.properties -jar
> /opt/graylog/server/graylog.jar server again and now I get the error:

> Any ideas?

you need to add as additional startup parameter to graylog!

as you use graylog OVA image i had created this issue:
https://github.com/Graylog2/omnibus-graylog2/issues/31

because this is not save possible.

/jd

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/CAGm-bLb4v0JHLz5acB2A6s6dYqH31fNUU_Y3OM8PVijFYhCD3w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Anyone use Image in real world application? Graylog 2.0 image fails after few days. Is this Image problem or Graylog in general?

2016-06-27 Thread John

1 and 4
and the graylog server node is not sending data to elasticsearch
I deleted the journal but it doesn't help
the problems began few days after I upgraded from 1.3 to 2.0.2

בתאריך יום שני, 27 ביוני 2016 בשעה 14:30:28 UTC+3, מאת Joe K:
>
> Which problem out of 4?
>
>
> On Monday, June 27, 2016 at 2:00:14 PM UTC+3, John wrote:
>>
>> Hi Joe
>> I have exactly the same problem few days after I upgraded from 1.3 to 
>> 2.0.2
>> Did you managed to fix this issue?
>>
>> בתאריך יום חמישי, 26 במאי 2016 בשעה 14:02:19 UTC+3, מאת Joe K:
>>>
>>>
>>> - We run it on t2.medium. (4GB RAM, 2 cores)
>>> - About 1 incoming message per second.
>>> - tried 2.0.0 and now running 2.0.1
>>>
>>> Anyone use Image in real world application? Graylog 2.0 image fails 
>>> after few days. Is this Image problem or Graylog in general?
>>>
>>> It runs fine for about a week. After that there's errors and search stop 
>>> working. Search requests timeout.
>>> There's many errors and they are very cryptic, google search does not 
>>> give any solutions how to manage them:
>>>
>>>
>>> *1. After about a week we have error "Uncommited messages deleted from 
>>> journal"*
>>>
 Uncommited messages deleted from journal (triggered 9 days ago)
 Some messages were deleted from the Graylog journal before they could 
 be written to Elasticsearch. Please verify that your Elasticsearch cluster 
 is healthy and fast enough. You may also want to review your Graylog 
 journal settings and set a higher limit. (Node: f12...
>>>
>>>
>>> What to do about this? What is "journal"? Google search produce no 
>>> answers.
>>>
>>> *2. After about 4 days of clean install it always trigger "Cluster 
>>> unhealthy"*
>>>
  "Elasticsearch cluster unhealthy (RED)"
 "The Elasticsearch cluster state is RED which means shards are 
 unassigned. This usually indicates a crashed and corrupt cluster and needs 
 to be investigated. Graylog will write into the local disk journal. Read 
 how to fix this in the Elasticsearch setup documentation."
>>>
>>>
>>> When you go to that documentation link it says "The red status indicates 
>>> that some or all of the primary shards are not available. In this state, no 
>>> searches can be performed until all primary shards are restored."
>>> That's it. what are you supposed to do?
>>> After long search finally found one solution: this was cured once with 
>>> *curl 
>>> -XPUT 'localhost:9200/_settings' -d '{ "index" : {   
>>>  "number_of_replicas" : 0}}'*
>>> Next time it happened, we tried the solution again, but response was 
>>> *{"acknowledged":false}*
>>> So what now???
>>>
>>> *3. Every time we perform graylog-ctl restart four more unassigled 
>>> shards appear:*
>>>  Elasticsearch cluster is yellow. Shards: 20 active, 0 initializing, 0
>>>  relocating, 8 unassigned
>>> graylog-ctl restart
>>>  Elasticsearch cluster is yellow. Shards: 20 active, 0 initializing, 0
>>>  relocating, 12 unassigned
>>> Etc.
>>>
>>>
>>>
>>> *4. Journal utilization is too high without any hint on how to set it to 
>>> higher.*
>>>
  Journal utilization is too high (triggered 11 days ago)
 Journal utilization is too high and may go over the limit soon. Please 
 verify that your Elasticsearch cluster is healthy and fast enough. You may 
 also want to review your Graylog journal settings and set a higher limit. 
 (Node: f121
>>>
>>>
>>> What is this "journal"? and how to set it to "higher"?
>>>
>>> Please help!
>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/2288cbf2-6f37-4e77-8c32-c50ba64fe71e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Anyone use Image in real world application? Graylog 2.0 image fails after few days. Is this Image problem or Graylog in general?

2016-06-27 Thread Joe K

Which problem out of 4?


On Monday, June 27, 2016 at 2:00:14 PM UTC+3, John wrote:
>
> Hi Joe
> I have exactly the same problem few days after I upgraded from 1.3 to 2.0.2
> Did you managed to fix this issue?
>
> בתאריך יום חמישי, 26 במאי 2016 בשעה 14:02:19 UTC+3, מאת Joe K:
>>
>>
>> - We run it on t2.medium. (4GB RAM, 2 cores)
>> - About 1 incoming message per second.
>> - tried 2.0.0 and now running 2.0.1
>>
>> Anyone use Image in real world application? Graylog 2.0 image fails after 
>> few days. Is this Image problem or Graylog in general?
>>
>> It runs fine for about a week. After that there's errors and search stop 
>> working. Search requests timeout.
>> There's many errors and they are very cryptic, google search does not 
>> give any solutions how to manage them:
>>
>>
>> *1. After about a week we have error "Uncommited messages deleted from 
>> journal"*
>>
>>> Uncommited messages deleted from journal (triggered 9 days ago)
>>> Some messages were deleted from the Graylog journal before they could be 
>>> written to Elasticsearch. Please verify that your Elasticsearch cluster is 
>>> healthy and fast enough. You may also want to review your Graylog journal 
>>> settings and set a higher limit. (Node: f12...
>>
>>
>> What to do about this? What is "journal"? Google search produce no 
>> answers.
>>
>> *2. After about 4 days of clean install it always trigger "Cluster 
>> unhealthy"*
>>
>>>  "Elasticsearch cluster unhealthy (RED)"
>>> "The Elasticsearch cluster state is RED which means shards are 
>>> unassigned. This usually indicates a crashed and corrupt cluster and needs 
>>> to be investigated. Graylog will write into the local disk journal. Read 
>>> how to fix this in the Elasticsearch setup documentation."
>>
>>
>> When you go to that documentation link it says "The red status indicates 
>> that some or all of the primary shards are not available. In this state, no 
>> searches can be performed until all primary shards are restored."
>> That's it. what are you supposed to do?
>> After long search finally found one solution: this was cured once with *curl 
>> -XPUT 'localhost:9200/_settings' -d '{ "index" : {   
>>  "number_of_replicas" : 0}}'*
>> Next time it happened, we tried the solution again, but response was 
>> *{"acknowledged":false}*
>> So what now???
>>
>> *3. Every time we perform graylog-ctl restart four more unassigled shards 
>> appear:*
>>  Elasticsearch cluster is yellow. Shards: 20 active, 0 initializing, 0
>>  relocating, 8 unassigned
>> graylog-ctl restart
>>  Elasticsearch cluster is yellow. Shards: 20 active, 0 initializing, 0
>>  relocating, 12 unassigned
>> Etc.
>>
>>
>>
>> *4. Journal utilization is too high without any hint on how to set it to 
>> higher.*
>>
>>>  Journal utilization is too high (triggered 11 days ago)
>>> Journal utilization is too high and may go over the limit soon. Please 
>>> verify that your Elasticsearch cluster is healthy and fast enough. You may 
>>> also want to review your Graylog journal settings and set a higher limit. 
>>> (Node: f121
>>
>>
>> What is this "journal"? and how to set it to "higher"?
>>
>> Please help!
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/31384281-e0df-46f8-8481-f97c4188cb10%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Having some difficulties with 3 node graylog cluster

2016-06-27 Thread Jan Doberstein

Hej Yiannis,




On 24. Juni 2016 at 16:19:01, Yiannis (k...@stoiximan.gr) wrote:
> the setup is really straight forward and never thought that i will have
> difficulties but….

you are faced a strange issue. That looks like a corner-case.

>
> On Friday, June 24, 2016 at 10:42:21 AM UTC+2, Jan Doberstein wrote:

> That is my starting papameters for all graylog server
> GRAYLOG_SERVER_JAVA_OPTS=
> "-Xms8g -Xmx8g -XX:NewRatio=1 -server -XX:+ResizeTLAB
> -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled
> -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC
> -XX:-OmitStackTraceInFastThrow"
>

> > > My 2 biggest problem are:
> > >
> > > 1) Most of the times when i press the search button (and only the search
> > > button displayed in the image)
> > >
> > > seems to me that my browser goes again from the login screen (to send
> > again
> > > the user credential) before rendering the results
> >
> > Can you please look into your log files of graylog when this happens
> > to you - it should be possible to get an idea why this happen just by
> > look at the log file during this ‘event’.

> When the log lever is INFO nothing appears in the log during this ‘event’,
> when i change to DEBUG or TRACE i really can't get the idea of what is
> happening.

if possible can you upload such logs somewhere - just because this is
something Graylog related and not Elasticsearch or Mongo.
This would be really helpful to help.



> > > 2) Every now and then, i get a strange error (when mostly when using
> > > firefox) from webs interface api server like the following
> > > (no errors on shown in the graylog server logs)
> >
> > Are you sure that you read
> >
> > http://docs.graylog.org/en/2.0/pages/configuration/web_interface.html#overview
> >
> > and set all Configurations to that?
> >
> > Even if you run the Web Interface only on one Node the API of all
> > Nodes need to be reachable by your browser.
> >
> >
> I believe i did

> and yes the API of all Nodes is reachable from my browser.

It could be that you are faced the issue Jason mentioned in his Mail -
maybe you can give us some Information that we are able to reproduce.

thx
Jan

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/CAGm-bLZWZfxDH4Yk9%3DzwsG7YE9f54z6Y%3DkkG32QaAdxBd6-wrQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Anyone use Image in real world application? Graylog 2.0 image fails after few days. Is this Image problem or Graylog in general?

2016-06-27 Thread John

Hi Joe
I have exactly the same problem few days after I upgraded from 1.3 to 2.0.2
Did you managed to fix this issue?

בתאריך יום חמישי, 26 במאי 2016 בשעה 14:02:19 UTC+3, מאת Joe K:
>
>
> - We run it on t2.medium. (4GB RAM, 2 cores)
> - About 1 incoming message per second.
> - tried 2.0.0 and now running 2.0.1
>
> Anyone use Image in real world application? Graylog 2.0 image fails after 
> few days. Is this Image problem or Graylog in general?
>
> It runs fine for about a week. After that there's errors and search stop 
> working. Search requests timeout.
> There's many errors and they are very cryptic, google search does not give 
> any solutions how to manage them:
>
>
> *1. After about a week we have error "Uncommited messages deleted from 
> journal"*
>
>> Uncommited messages deleted from journal (triggered 9 days ago)
>> Some messages were deleted from the Graylog journal before they could be 
>> written to Elasticsearch. Please verify that your Elasticsearch cluster is 
>> healthy and fast enough. You may also want to review your Graylog journal 
>> settings and set a higher limit. (Node: f12...
>
>
> What to do about this? What is "journal"? Google search produce no answers.
>
> *2. After about 4 days of clean install it always trigger "Cluster 
> unhealthy"*
>
>>  "Elasticsearch cluster unhealthy (RED)"
>> "The Elasticsearch cluster state is RED which means shards are 
>> unassigned. This usually indicates a crashed and corrupt cluster and needs 
>> to be investigated. Graylog will write into the local disk journal. Read 
>> how to fix this in the Elasticsearch setup documentation."
>
>
> When you go to that documentation link it says "The red status indicates 
> that some or all of the primary shards are not available. In this state, no 
> searches can be performed until all primary shards are restored."
> That's it. what are you supposed to do?
> After long search finally found one solution: this was cured once with *curl 
> -XPUT 'localhost:9200/_settings' -d '{ "index" : {   
>  "number_of_replicas" : 0}}'*
> Next time it happened, we tried the solution again, but response was 
> *{"acknowledged":false}*
> So what now???
>
> *3. Every time we perform graylog-ctl restart four more unassigled shards 
> appear:*
>  Elasticsearch cluster is yellow. Shards: 20 active, 0 initializing, 0
>  relocating, 8 unassigned
> graylog-ctl restart
>  Elasticsearch cluster is yellow. Shards: 20 active, 0 initializing, 0
>  relocating, 12 unassigned
> Etc.
>
>
>
> *4. Journal utilization is too high without any hint on how to set it to 
> higher.*
>
>>  Journal utilization is too high (triggered 11 days ago)
>> Journal utilization is too high and may go over the limit soon. Please 
>> verify that your Elasticsearch cluster is healthy and fast enough. You may 
>> also want to review your Graylog journal settings and set a higher limit. 
>> (Node: f121
>
>
> What is this "journal"? and how to set it to "higher"?
>
> Please help!
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/8fde5052-0f72-4740-bfa2-3e12a47b3d52%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Graylog wont send Data to Elasticsearch after Update

2016-06-27 Thread John

Hi,
I have the same problem after upgrade from 1.3 to 2.0.2
with 2 error messages "Uncommited messages deleted from journal"
and "
Journal utilization is too high"
and I deleted the journal from all of the nodes .but it doesn't help. I 
still have the same error messages and Graylog server is not sending data 
to elastic search for 5 days.
The nodes only write the logs to the journal
Thx for your help

logs from graylog server

2016-06-27_10:43:57.18991   at 
org.graylog2.buffers.processors.OutputBufferProcessor$1.run(OutputBufferProcessor.java:189)
 
[graylog.jar:?]
2016-06-27_10:43:57.19030   at 
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
 
[graylog.jar:?]
2016-06-27_10:43:57.19153   at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
[?:1.8.0_77]
2016-06-27_10:43:57.19206   at 
java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_77]
2016-06-27_10:43:57.19262   at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[?:1.8.0_77]
2016-06-27_10:43:57.19423   at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[?:1.8.0_77]
2016-06-27_10:43:57.19489   at java.lang.Thread.run(Thread.java:745) 
[?:1.8.0_77]
2016-06-27_10:43:57.23458 2016-06-27 13:43:57,234 ERROR: 
org.graylog2.outputs.BlockingBatchedESOutput - Unable to flush message 
buffer
2016-06-27_10:43:57.23517 java.lang.ClassCastException: Cannot cast 
java.lang.Integer to java.lang.String
2016-06-27_10:43:57.23653   at java.lang.Class.cast(Class.java:3369) 
~[?:1.8.0_77]
2016-06-27_10:43:57.23702   at 
org.graylog2.plugin.Message.getFieldAs(Message.java:370) ~[graylog.jar:?]
2016-06-27_10:43:57.23862   at 
org.graylog2.plugin.Message.getMessage(Message.java:286) ~[graylog.jar:?]
2016-06-27_10:43:57.23938   at 
org.graylog2.plugin.Message.toElasticSearchObject(Message.java:215) 
~[graylog.jar:?]
2016-06-27_10:43:57.24141   at 
org.graylog2.indexer.messages.Messages.bulkIndex(Messages.java:116) 
~[graylog.jar:?]
2016-06-27_10:43:57.24205   at 
org.graylog2.indexer.messages.Messages.bulkIndex(Messages.java:106) 
~[graylog.jar:?]
2016-06-27_10:43:57.24275   at 
org.graylog2.outputs.ElasticSearchOutput.write(ElasticSearchOutput.java:98) 
~[graylog.jar:?]
2016-06-27_10:43:57.24393   at 
org.graylog2.outputs.BlockingBatchedESOutput.flush(BlockingBatchedESOutput.java:128)
 
[graylog.jar:?]
2016-06-27_10:43:57.24593   at 
org.graylog2.outputs.BlockingBatchedESOutput.write(BlockingBatchedESOutput.java:105)
 
[graylog.jar:?]
2016-06-27_10:43:57.24638   at 
org.graylog2.buffers.processors.OutputBufferProcessor$1.run(OutputBufferProcessor.java:189)
 
[graylog.jar:?]
2016-06-27_10:43:57.24733   at 
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
 
[graylog.jar:?]
2016-06-27_10:43:57.24855   at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
[?:1.8.0_77]
2016-06-27_10:43:57.24937   at 
java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_77]
2016-06-27_10:43:57.25088   at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[?:1.8.0_77]
2016-06-27_10:43:57.25162   at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[?:1.8.0_77]
2016-06-27_10:43:57.25262   at java.lang.Thread.run(Thread.java:745) 
[?:1.8.0_77]
2016-06-27_10:43:57.28833 2016-06-27 13:43:57,287 ERROR: 
org.graylog2.outputs.BlockingBatchedESOutput - Unable to flush message 
buffer
2016-06-27_10:43:57.28870 java.lang.ClassCastException: Cannot cast 
java.lang.Integer to java.lang.String
2016-06-27_10:43:57.28948   at java.lang.Class.cast(Class.java:3369) 
~[?:1.8.0_77]
2016-06-27_10:43:57.29100   at 
org.graylog2.plugin.Message.getFieldAs(Message.java:370) ~[graylog.jar:?]
2016-06-27_10:43:57.29180   at 
org.graylog2.plugin.Message.getMessage(Message.java:286) ~[graylog.jar:?]
2016-06-27_10:43:57.29325   at 
org.graylog2.plugin.Message.toElasticSearchObject(Message.java:215) 
~[graylog.jar:?]
2016-06-27_10:43:57.29386   at 
org.graylog2.indexer.messages.Messages.bulkIndex(Messages.java:116) 
~[graylog.jar:?]
2016-06-27_10:43:57.29528   at 
org.graylog2.indexer.messages.Messages.bulkIndex(Messages.java:106) 
~[graylog.jar:?]
2016-06-27_10:43:57.29588   at 
org.graylog2.outputs.ElasticSearchOutput.write(ElasticSearchOutput.java:98) 
~[graylog.jar:?]
2016-06-27_10:43:57.29712   at 
org.graylog2.outputs.BlockingBatchedESOutput.flush(BlockingBatchedESOutput.java:128)
 
[graylog.jar:?]
2016-06-27_10:43:57.29765   at 
org.graylog2.outputs.BlockingBatchedESOutput.write(BlockingBatchedESOutput.java:105)
 
[graylog.jar:?]
2016-06-27_10:43:57.29874   at 
org.graylog2.buffers.processors.OutputBufferProcessor$1.run(OutputBufferProcessor.java:189)

[graylog2] Graylog wont send Data to Elasticsearch after Update

2016-06-27 Thread toni . frommknecht

Hi,

today we updated Graylog to 2.0.3 and scince that no more data were send to 
elasticsearch.
We recieve data but dont write them to elasticsearch
Any idea why this could happen?

Greetings!





Graylog:
2016-06-27T10:26:43.744+02:00 INFO  [CmdLineTool] Loaded plugin: Elastic 
Beats Input 1.0.1 [org.graylog.plugins.beats.BeatsInputPlugin]
2016-06-27T10:26:43.745+02:00 INFO  [CmdLineTool] Loaded plugin: Collector 
1.0.2 [org.graylog.plugins.collector.CollectorPlugin]
2016-06-27T10:26:43.745+02:00 INFO  [CmdLineTool] Loaded plugin: Enterprise 
Integration Plugin 1.0.2 
[org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]
2016-06-27T10:26:43.745+02:00 INFO  [CmdLineTool] Loaded plugin: 
MapWidgetPlugin 1.0.2 [org.graylog.plugins.map.MapWidgetPlugin]
2016-06-27T10:26:43.746+02:00 INFO  [CmdLineTool] Loaded plugin: Pipeline 
Processor Plugin 1.0.0-beta.4 
[org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2016-06-27T10:26:43.746+02:00 INFO  [CmdLineTool] Loaded plugin: Anonymous 
Usage Statistics 2.0.2 
[org.graylog.plugins.usagestatistics.UsageStatsPlugin]
2016-06-27T10:26:43.856+02:00 INFO  [CmdLineTool] Running with JVM 
arguments: -Xms1g -Xmx2g -XX:NewRatio=1 -XX:+ResizeTLAB 
-XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled 
-XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC 
-XX:-OmitStackTraceInFastThrow 
-Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml 
-Djava.library.path=/usr/share/graylog-server/lib/sigar 
-Dgraylog2.installation_source=rpm
2016-06-27T10:26:46.138+02:00 INFO  [InputBufferImpl] Message journal is 
enabled.
2016-06-27T10:26:46.348+02:00 INFO  [LogManager] Loading logs.
2016-06-27T10:26:46.525+02:00 INFO  [LogManager] Logs loading complete.
2016-06-27T10:26:46.526+02:00 INFO  [KafkaJournal] Initialized Kafka based 
journal at /var/lib/graylog-server/journal
2016-06-27T10:26:46.541+02:00 INFO  [InputBufferImpl] Initialized 
InputBufferImpl with ring size <65536> and wait strategy 
, running 2 parallel message handlers.
2016-06-27T10:26:46.581+02:00 INFO  [cluster] Cluster created with settings 
{hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, 
serverSelectionTimeout='3 ms', maxWaitQueueSize=5000}
2016-06-27T10:26:46.615+02:00 INFO  [cluster] No server chosen by 
ReadPreferenceServerSelector{readPreference=primary} from cluster 
description ClusterDescription{type=UNKNOWN, connectionMode=SINGLE, 
all=[ServerDescription{address=localhost:27017, type=UNKNOWN, 
state=CONNECTING}]}. Waiting for 3 ms before timing out
2016-06-27T10:26:46.653+02:00 INFO  [connection] Opened connection 
[connectionId{localValue:1, serverValue:36}] to localhost:27017
2016-06-27T10:26:46.655+02:00 INFO  [cluster] Monitor thread successfully 
connected to server with description 
ServerDescription{address=localhost:27017, type=STANDALONE, 
state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 2, 7]}, 
minWireVersion=0, maxWireVersion=4, maxDocumentSize=16777216, 
roundTripTimeNanos=736400}
2016-06-27T10:26:46.661+02:00 INFO  [connection] Opened connection 
[connectionId{localValue:2, serverValue:37}] to localhost:27017
2016-06-27T10:26:46.903+02:00 INFO  [NodeId] Node ID: 
006e89e5-73c8-46dd-ac86-7c1ddb26ed84
2016-06-27T10:26:47.015+02:00 INFO  [node] 
[graylog-006e89e5-73c8-46dd-ac86-7c1ddb26ed84] version[2.3.2], pid[3533], 
build[b9e4a6a/2016-04-21T16:03:47Z]
2016-06-27T10:26:47.015+02:00 INFO  [node] 
[graylog-006e89e5-73c8-46dd-ac86-7c1ddb26ed84] initializing ...
2016-06-27T10:26:47.022+02:00 INFO  [plugins] 
[graylog-006e89e5-73c8-46dd-ac86-7c1ddb26ed84] modules [], plugins 
[graylog-monitor], sites []
2016-06-27T10:26:48.829+02:00 INFO  [node] 
[graylog-006e89e5-73c8-46dd-ac86-7c1ddb26ed84] initialized
2016-06-27T10:26:48.904+02:00 INFO  [Version] HV01: Hibernate Validator 
5.2.4.Final
2016-06-27T10:26:49.054+02:00 INFO  [ProcessBuffer] Initialized 
ProcessBuffer with ring size <65536> and wait strategy 
.
2016-06-27T10:26:51.042+02:00 INFO  [RulesEngineProvider] No static rules 
file loaded.
2016-06-27T10:26:51.085+02:00 INFO  [connection] Opened connection 
[connectionId{localValue:3, serverValue:38}] to localhost:27017
2016-06-27T10:26:51.192+02:00 WARN  [GeoIpResolverEngine] GeoIP database 
file does not exist: /tmp/GeoLite2-City.mmdb
2016-06-27T10:26:51.198+02:00 INFO  [OutputBuffer] Initialized OutputBuffer 
with ring size <65536> and wait strategy .
2016-06-27T10:26:51.242+02:00 WARN  [GeoIpResolverEngine] GeoIP database 
file does not exist: /tmp/GeoLite2-City.mmdb
2016-06-27T10:26:51.285+02:00 WARN  [GeoIpResolverEngine] GeoIP database 
file does not exist: /tmp/GeoLite2-City.mmdb
2016-06-27T10:26:51.327+02:00 WARN  [GeoIpResolverEngine] GeoIP database 
file does not exist: /tmp/GeoLite2-City.mmdb
2016-06-27T10:26:51.362+02:00 WARN  [GeoIpResolverEngine] GeoIP database 
file does not exist: /tmp/GeoLite2-City.mmdb
2016-06-27T10:26:51.864+02:00 INFO  [ServerBootstrap] Graylog server 2.0.2 
(4da1379) starting up

[graylog2] Colors in Charts and Data Table

[graylog2] Adding Custom Functions for defining Rules

[graylog2] Kafka output plugin for Graylog 2.x?

Re: [graylog2] Having some difficulties with 3 node graylog cluster

[graylog2] Alerting for flopping ports

[graylog2] Backup of indices in Graylog 1.3

Re: [graylog2] Need some help disabling ciphers and algorithms

Re: [graylog2] Need some help disabling ciphers and algorithms

Re: [graylog2] Need some help disabling ciphers and algorithms

Re: [graylog2] Re: Anyone use Image in real world application? Graylog 2.0 image fails after few days. Is this Image problem or Graylog in general?

[graylog2] Syslog messages look different between Splunk and Graylog

Re: [graylog2] Need some help disabling ciphers and algorithms

Re: [graylog2] Re: Anyone use Image in real world application? Graylog 2.0 image fails after few days. Is this Image problem or Graylog in general?

Re: [graylog2] Re: Anyone use Image in real world application? Graylog 2.0 image fails after few days. Is this Image problem or Graylog in general?

Re: [graylog2] Need some help disabling ciphers and algorithms

[graylog2] Re: Anyone use Image in real world application? Graylog 2.0 image fails after few days. Is this Image problem or Graylog in general?

[graylog2] Re: Anyone use Image in real world application? Graylog 2.0 image fails after few days. Is this Image problem or Graylog in general?

Re: [graylog2] Having some difficulties with 3 node graylog cluster

[graylog2] Re: Anyone use Image in real world application? Graylog 2.0 image fails after few days. Is this Image problem or Graylog in general?

[graylog2] Re: Graylog wont send Data to Elasticsearch after Update

[graylog2] Graylog wont send Data to Elasticsearch after Update

21 matches

Site Navigation

Mail list logo

Footer information