[graylog2] Incoming Gelf UDP messages not showing up

[IJFK]

Hi,

New Graylog user here. Bit of a dilemma, been trying to figure this out for 
hours now without success, and about to give up. I'm using their appliance 
which I just downloaded yesterday.

I'm sending Syslog packets in Gelf format (I successfully validated the 
Json), and no matter what I do, the packets don't show up. There is no 
parsing error or anything, the data just doesn't show up.

I already created a Raw/UDP input & stream, which does show the messages 
coming in, I also verified with tcpdump that they are actually making it to 
the server.

What's really strange is that sending a traditional UDP Syslog RFC 3164 
packet to port 12201 will yield in a parsing error (since it's not json), 
suggesting that it's definitely getting data from that port, I'm assuming 
the stream is working to a certain extent?

However I cannot figure out what is happening to those other messages that 
should be coming in and get parsed. This worked at some point, and then 
just stopped working randomly. Tried to reboot the VM, no success. I'm 
currently even trying to send the standard test JSON 
(from http://docs.graylog.org/en/2.1/pages/gelf.html) without success.

Any suggestions would be appreciate, thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/c73546c6-a619-4541-9f92-15a5be505662%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Graylog is ignoring some UDP packets sent by a particular host

[IJFK]

I'm having similar issues with GELF packets. They show up if I create a raw 
udp input, but they don't show up with a gelf input. It used to work, but 
suddenly stopped working.

I also have no idea on how to debug this, there doesn't seem to be a place 
for parser errors. Increasing the debug level to "debug" or "trace" doesn't 
help me, it generates way too much noise.

On Wednesday, February 8, 2017 at 12:43:38 PM UTC-6, tomaszik...@gmail.com 
wrote:
>
> Hello,
>
> I've recently set up a working Graylog server. It's collecting logs from 
> many network switches and routers. One particular router (ironically, the 
> most important one) doesn't appear in the Sources list though. Graylog 
> keeps ignoring all packets coming from that host. Here's an example of a 
> packet which is *not* ignored by Graylog:
>
> 19:12:15.705167 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto 
> UDP (17), length 115)
> 10.50.255.44.40810 > Silenoz.syslog: [udp sum ok] [|syslog]
>  0x:  4500 0073  4000 4011 27e3 0a32 ff2c  E..s..@.@.'..2.,
>  0x0010:  0a32 ff06 9f6a 0202 005f 01d1 6468 6370  .2...j..._..dhcp
>  0x0020:  2c77 6172 6e69 6e67 2067 706f 6e2d 6d6e  ,warning.gpon-mn
>  0x0030:  6720 6f66 6665 7269 6e67 206c 6561 7365  g.offering.lease
>  0x0040:  2031 302e 3530 2e32 3338 2e33 3520 666f  .10.50.238.35.fo
>  0x0050:  7220 3030 3a30 323a 3731 3a35 413a 3036  r.00:02:71:5A:06
>  0x0060:  3a42 3820 7769 7468 6f75 7420 7375 6363  :B8.without.succ
>  0x0070:  6573 73 
>
> And below you can see a packet which *is* ignored by Graylog:
>
> 10.50.255.111.56993 > Silenoz.syslog: [udp sum ok] SYSLOG, length: 154
>  Facility local7 (23), Severity notice (5)
>  Msg: Feb 8 19:12:17: %SYSLOG-5-NOTICE: aaad: SubSessionAUTHFAIL user: 
> pppoe16344@mn (24) Authentication failure [Circuit handle: 1/4:511:63:31/6
> /2/47661]\0x0a
>  0x:  3c31 3839 3e46 6562 2038 2031 393a 3132
>  0x0010:  3a31 373a 2025 5359 534c 4f47 2d35 2d4e
>  0x0020:  4f54 4943 453a 2061 6161 643a 2053 7562
>  0x0030:  5365 7373 696f 6e41 5554 4846 4149 4c20
>  0x0040:  7573 6572 3a20 7070 706f 6531 3633 3434
>  0x0050:  406d 6e20 2832 3429 2041 7574 6865 6e74
>  0x0060:  6963 6174 696f 6e20 6661 696c 7572 6520
>  0x0070:  5b43 6972 6375 6974 2068 616e 646c 653a
>  0x0080:  2031 2f34 3a35 3131 3a36 333a 3331 2f36
>  0x0090:  2f32 2f34 3736 3631 5d0a
>  0x:  4500 00b6 77da  4011 ef82 0a32 ff6f  E...w...@2.o
>  0x0010:  0a32 ff06 dea1 0202 00a2 28d8 3c31 3839  .2(.<189
>  0x0020:  3e46 6562 2038 2031 393a 3132 3a31 373a  >Feb.8.19:12:17:
>  0x0030:  2025 5359 534c 4f47 2d35 2d4e 4f54 4943  .%SYSLOG-5-NOTIC
>  0x0040:  453a 2061 6161 643a 2053 7562 5365 7373  E:.aaad:.SubSess
>  0x0050:  696f 6e41 5554 4846 4149 4c20 7573 6572  ionAUTHFAIL.user
>  0x0060:  3a20 7070 706f 6531 3633 3434 406d 6e20  :.pppoe16344@mn.
>  0x0070:  2832 3429 2041 7574 6865 6e74 6963 6174  (24).Authenticat
>  0x0080:  696f 6e20 6661 696c 7572 6520 5b43 6972  ion.failure.[Cir
>  0x0090:  6375 6974 2068 616e 646c 653a 2031 2f34  cuit.handle:.1/4
>  0x00a0:  3a35 3131 3a36 333a 3331 2f36 2f32 2f34  :511:63:31/6/2/4
>  0x00b0:  3736 3631 5d0a   7661].
>
> As you can see, the packet is much longer, but it doesn't exceed the 
> maximum UDP packet size that can be processed by Graylog (8192). My guess 
> is that logs coming from 10.50.255.111 are not RFC compatible and thus 
> they're discarded by Graylog. How can I debug it / fix it? I didn't find 
> any related messages in the Elasticsearch log (there were no errors related 
> to parsing a message).
> I deleted the default Input object and added a new RAW UDP Input object. 
> It didn't fix the issue - logs from 10.50.255.111 are still not parsed.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/e716e116-e7dc-4f8c-a032-8f06b53ac692%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] collect logs from remote machine

[Wallace Turner]

The graylog homepage state *"No more logging into multiple devices to parse 
plain text log files."*
but I am yet unable to figure out how it does this.
The docs located at http://docs.graylog.org/en/2.1/pages/sending_data.html# 
go thru many steps but none in which the graylog server/process will log 
into servers and parse a log file.

What i am trying to do is for graylog to retrieve (or monitor) a log file 
at a network location (windows servers) and bring the contents of the plain 
text log file to graylog.

Is this possible (on windows)?

Ideally it wouldnt bring down the entire file each time but could determine 
if the file has changed and only download the relevant part

Wal

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/785f0619-b042-4c75-bf06-07e9cb2da4ba%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Forward from One graylog to another

[Tom Powers]

Is there any good doc on setting up the tls on the stream output and then the 
receiving side at the new graylog instance?

Been combing through doc and posts for a couple hours and only have fragments 
of an idea on how to do this

Self signed certs will be fine for this 

All insight is appreciated

Tp

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/36298fef-575c-4a60-8b4a-61759677b296%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Forward from One graylog to another

[Tom Powers]

Is there any good doc on setting up the tls on the stream output and then the 
receiving side at the new graylog instance?

Been combing through doc and posts for a couple hours and only have fragments 
of an idea on how to do this

Self signed certs will be fine for this 

All insight is appreciated

Tp

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/06fa8cdf-e4b5-4509-bebe-9d3c7fc4ca53%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Forward from One graylog to another

[Jochen Schalanda]

Hi Tom,

On Wednesday, 8 February 2017 23:31:46 UTC+1, Tom Powers wrote:
>
> We are only tracking windows events here, so If I read this right, could i 
> set the stream output in Gelf format and send it to the Parent office 
> Graylog server (over TLS of course)?
>

Yes, that's pretty much it. Assign a GELF output to the relevant streams 
and send them to your central Graylog instance (with a GELF input).

Cheers,
Jochen 

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/95b47bab-accd-4bc5-93b1-512e8c869151%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Forward from One graylog to another

[Tom Powers]

I have 2 sites.  One office is the main office, the other is a branch office

I am wondering if this is possible.

If I put a graylog server at each site in regular setup, I can collect the 
logs of that site. Simple enough so far.

Now...the Streams I have setup on those 2 servers, which is polling the 
events I really care about,  is there any way to get the streams to forward 
those matching events to a 3rd graylog server at our parent office?  That 
way, the parent office only sees the info that the streams grabbed and 
nothing else.

We are only tracking windows events here, so If I read this right, could i 
set the stream output in Gelf format and send it to the Parent office 
Graylog server (over TLS of course)?

All insight is appreciated

Thanks

TP

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/e97dedc2-6a3b-4435-97f5-1eb42075564e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Overwriting Timestamp field using Pipeline rules

[Al Reynolds]

I've noticed another error. The timestamp field is being replaced 
correctly, but the "gl2_processing_error" field is showing the following 
error (on all messages):
For rule 'WO-CS-RAS': In call to function 'parse_date' at 8:15 an exception 
was thrown: Invalid format: "2017-02-08 15:05:59,170" is malformed at "17-02-08 
15:05:59,170"


It doesn't seem to have any adverse effects, but I'm curious as to what 
might be causing it?

On Wednesday, February 8, 2017 at 1:56:17 PM UTC-5, Al Reynolds wrote:
>
> Figured it out--parse_date needed the timestamp . New rule looks like this:
> rule "WO-CS-RAS" 
> when 
> 
> contains(to_string($message.file),"centralserver\\ras-server\\log\\ras_cs_")
> then
> set_field("WO_Log_Source","RAS-CS");
> let matches = grok(pattern: "%{WO_CS_RAS_CS_MESSAGE}", value: 
> to_string($message.message));
> set_fields(matches);
> let date = parse_date(to_string($message.WO_Timestamp), "-MM-dd 
> HH:mm:ss,SSS", "EST");
> set_field("timestamp", date);
> route_to_stream("WideOrbit Logs");
> end
>
> I was under the impression that the timezone was optional? 
>
> Thanks for all your help with this Jochen--it's greatly appreciated!
>
> Cheers,
> Al
>
> On Wednesday, February 8, 2017 at 11:05:22 AM UTC-5, Al Reynolds wrote:
>>
>> That's what I get for typing it out...thank you for catching that! 
>> Unfortunately, even after correcting for the incorrect milliseconds value, 
>> it's still not replacing timestamp value. I sent the parsed date to a new 
>> field (in this case, "log_timestamp") to verify that the output data was in 
>> the correct format, which it is now, but it still won't replace the 
>> timestamp field.
>>
>> Message sample with "log_timestamp" field:
>> WO_CS_RAS_CS_MESSAGE
>> 2017-02-08 11:00:34,980 WARN  [Task 'ATLANTA-FS' FS timer.1] 
>> FriendshipTasksServiceImpl = Could not obtain task info for:  
>> 2c95ac8e-57e3-91b2-0158-495b880b24e8REQUEST FAILED ==> STATUS CODE: 404, 
>> RESPONSE BODY:
>> WO_LogLevel
>> WARN
>> WO_Log_Source
>> RAS-CS
>> WO_Message
>> Could not obtain task info for:  2c95ac8e-57e3-91b2-0158-
>> 495b880b24e8REQUEST FAILED ==> STATUS CODE: 404, RESPONSE BODY:
>> WO_Process
>> Task 'ATLANTA-FS' FS timer.1
>> WO_SubProcess
>> FriendshipTasksServiceImpl
>> WO_Timestamp
>> 2017-02-08 11:00:34,980
>> facility
>> filebeat
>> file
>> d:\centralserver\ras-server\log\ras_cs_WO-ATL-CS.log
>> input_type
>> log
>> log_timestamp
>> 2017-02-08T11:00:34.980Z
>> message
>> 2017-02-08 11:00:34,980 WARN  [Task 'ATLANTA-FS' FS timer.1] 
>> FriendshipTasksServiceImpl = Could not obtain task info for:  
>> 2c95ac8e-57e3-91b2-0158-495b880b24e8REQUEST FAILED ==> STATUS CODE: 404, 
>> RESPONSE BODY:
>> name
>> WO-ATL-CS
>> offset
>> 2372156
>> source
>> WO-ATL-CS
>> timestamp
>> 2017-02-08T16:00:35.864Z
>> type
>> log
>>
>> Corrected rule: 
>> rule "WO-CS-RAS" 
>> when 
>> 
>> contains(to_string($message.file),"centralserver\\ras-server\\log\\ras_cs_")
>> then
>> set_field("WO_Log_Source","RAS-CS");
>> let matches = grok(pattern: "%{WO_CS_RAS_CS_MESSAGE}", value: 
>> to_string($message.message));
>> set_fields(matches);
>> let date = parse_date(to_string($message.WO_Timestamp), "-MM-dd 
>> HH:mm:ss,SSS");
>> set_field("timestamp", date);
>> route_to_stream("WideOrbit Logs");
>> end
>>
>> Thanks!
>>
>> Cheers,
>> Al
>>
>> On Wednesday, February 8, 2017 at 10:55:03 AM UTC-5, Jochen Schalanda 
>> wrote:
>>>
>>> Hi Al,
>>>
>>> On Wednesday, 8 February 2017 15:46:07 UTC+1, Al Reynolds wrote:

 WO_Timestamp
 2017-02-08 09:42:30,056

 Those messages are with the date parsing disabled. I'm attempting to 
 replace "timestamp" with the "WO_Timestamp" field. 

>>>
>>> The string in WO_Timestamp doesn't match the pattern "-MM-dd 
>>> HH:mm:ss,sss" used in parse_date(). See 
>>> http://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html
>>>  
>>> for details.
>>>
>>> Hint: 's' and 'S' are not the same thing.
>>>  
>>>
>>> Side note: The full_message field is empty on my filebeat inputs--is 
 that expected behavior? 

>>>
>>> Yes, that's expected.
>>>
>>> What would you expect to find in the (optional) full_message field?
>>>
>>> Cheers,
>>> Jochen
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/34c557b5-8552-405b-8892-04c8eb1ad2b1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Overwriting Timestamp field using Pipeline rules

[Al Reynolds]

Figured it out--parse_date needed the timestamp . New rule looks like this:
rule "WO-CS-RAS" 
when 

contains(to_string($message.file),"centralserver\\ras-server\\log\\ras_cs_")
then
set_field("WO_Log_Source","RAS-CS");
let matches = grok(pattern: "%{WO_CS_RAS_CS_MESSAGE}", value: 
to_string($message.message));
set_fields(matches);
let date = parse_date(to_string($message.WO_Timestamp), "-MM-dd 
HH:mm:ss,SSS", "EST");
set_field("timestamp", date);
route_to_stream("WideOrbit Logs");
end

I was under the impression that the timezone was optional? 

Thanks for all your help with this Jochen--it's greatly appreciated!

Cheers,
Al

On Wednesday, February 8, 2017 at 11:05:22 AM UTC-5, Al Reynolds wrote:
>
> That's what I get for typing it out...thank you for catching that! 
> Unfortunately, even after correcting for the incorrect milliseconds value, 
> it's still not replacing timestamp value. I sent the parsed date to a new 
> field (in this case, "log_timestamp") to verify that the output data was in 
> the correct format, which it is now, but it still won't replace the 
> timestamp field.
>
> Message sample with "log_timestamp" field:
> WO_CS_RAS_CS_MESSAGE
> 2017-02-08 11:00:34,980 WARN  [Task 'ATLANTA-FS' FS timer.1] 
> FriendshipTasksServiceImpl = Could not obtain task info for:  
> 2c95ac8e-57e3-91b2-0158-495b880b24e8REQUEST FAILED ==> STATUS CODE: 404, 
> RESPONSE BODY:
> WO_LogLevel
> WARN
> WO_Log_Source
> RAS-CS
> WO_Message
> Could not obtain task info for:  2c95ac8e-57e3-91b2-0158-
> 495b880b24e8REQUEST FAILED ==> STATUS CODE: 404, RESPONSE BODY:
> WO_Process
> Task 'ATLANTA-FS' FS timer.1
> WO_SubProcess
> FriendshipTasksServiceImpl
> WO_Timestamp
> 2017-02-08 11:00:34,980
> facility
> filebeat
> file
> d:\centralserver\ras-server\log\ras_cs_WO-ATL-CS.log
> input_type
> log
> log_timestamp
> 2017-02-08T11:00:34.980Z
> message
> 2017-02-08 11:00:34,980 WARN  [Task 'ATLANTA-FS' FS timer.1] 
> FriendshipTasksServiceImpl = Could not obtain task info for:  
> 2c95ac8e-57e3-91b2-0158-495b880b24e8REQUEST FAILED ==> STATUS CODE: 404, 
> RESPONSE BODY:
> name
> WO-ATL-CS
> offset
> 2372156
> source
> WO-ATL-CS
> timestamp
> 2017-02-08T16:00:35.864Z
> type
> log
>
> Corrected rule: 
> rule "WO-CS-RAS" 
> when 
> 
> contains(to_string($message.file),"centralserver\\ras-server\\log\\ras_cs_")
> then
> set_field("WO_Log_Source","RAS-CS");
> let matches = grok(pattern: "%{WO_CS_RAS_CS_MESSAGE}", value: 
> to_string($message.message));
> set_fields(matches);
> let date = parse_date(to_string($message.WO_Timestamp), "-MM-dd 
> HH:mm:ss,SSS");
> set_field("timestamp", date);
> route_to_stream("WideOrbit Logs");
> end
>
> Thanks!
>
> Cheers,
> Al
>
> On Wednesday, February 8, 2017 at 10:55:03 AM UTC-5, Jochen Schalanda 
> wrote:
>>
>> Hi Al,
>>
>> On Wednesday, 8 February 2017 15:46:07 UTC+1, Al Reynolds wrote:
>>>
>>> WO_Timestamp
>>> 2017-02-08 09:42:30,056
>>>
>>> Those messages are with the date parsing disabled. I'm attempting to 
>>> replace "timestamp" with the "WO_Timestamp" field. 
>>>
>>
>> The string in WO_Timestamp doesn't match the pattern "-MM-dd 
>> HH:mm:ss,sss" used in parse_date(). See 
>> http://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html
>>  
>> for details.
>>
>> Hint: 's' and 'S' are not the same thing.
>>  
>>
>> Side note: The full_message field is empty on my filebeat inputs--is that 
>>> expected behavior? 
>>>
>>
>> Yes, that's expected.
>>
>> What would you expect to find in the (optional) full_message field?
>>
>> Cheers,
>> Jochen
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/a3b908a8-4993-4107-87f8-55445605cc66%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Graylog is ignoring some UDP packets sent by a particular host

[tomaszikasperczyk]

Hello,

I've recently set up a working Graylog server. It's collecting logs from 
many network switches and routers. One particular router (ironically, the 
most important one) doesn't appear in the Sources list though. Graylog 
keeps ignoring all packets coming from that host. Here's an example of a 
packet which is *not* ignored by Graylog:

19:12:15.705167 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (
17), length 115)
10.50.255.44.40810 > Silenoz.syslog: [udp sum ok] [|syslog]
 0x:  4500 0073  4000 4011 27e3 0a32 ff2c  E..s..@.@.'..2.,
 0x0010:  0a32 ff06 9f6a 0202 005f 01d1 6468 6370  .2...j..._..dhcp
 0x0020:  2c77 6172 6e69 6e67 2067 706f 6e2d 6d6e  ,warning.gpon-mn
 0x0030:  6720 6f66 6665 7269 6e67 206c 6561 7365  g.offering.lease
 0x0040:  2031 302e 3530 2e32 3338 2e33 3520 666f  .10.50.238.35.fo
 0x0050:  7220 3030 3a30 323a 3731 3a35 413a 3036  r.00:02:71:5A:06
 0x0060:  3a42 3820 7769 7468 6f75 7420 7375 6363  :B8.without.succ
 0x0070:  6573 73 

And below you can see a packet which *is* ignored by Graylog:

10.50.255.111.56993 > Silenoz.syslog: [udp sum ok] SYSLOG, length: 154
 Facility local7 (23), Severity notice (5)
 Msg: Feb 8 19:12:17: %SYSLOG-5-NOTICE: aaad: SubSessionAUTHFAIL user: 
pppoe16344@mn (24) Authentication failure [Circuit handle: 1/4:511:63:31/6/2
/47661]\0x0a
 0x:  3c31 3839 3e46 6562 2038 2031 393a 3132
 0x0010:  3a31 373a 2025 5359 534c 4f47 2d35 2d4e
 0x0020:  4f54 4943 453a 2061 6161 643a 2053 7562
 0x0030:  5365 7373 696f 6e41 5554 4846 4149 4c20
 0x0040:  7573 6572 3a20 7070 706f 6531 3633 3434
 0x0050:  406d 6e20 2832 3429 2041 7574 6865 6e74
 0x0060:  6963 6174 696f 6e20 6661 696c 7572 6520
 0x0070:  5b43 6972 6375 6974 2068 616e 646c 653a
 0x0080:  2031 2f34 3a35 3131 3a36 333a 3331 2f36
 0x0090:  2f32 2f34 3736 3631 5d0a
 0x:  4500 00b6 77da  4011 ef82 0a32 ff6f  E...w...@2.o
 0x0010:  0a32 ff06 dea1 0202 00a2 28d8 3c31 3839  .2(.<189
 0x0020:  3e46 6562 2038 2031 393a 3132 3a31 373a  >Feb.8.19:12:17:
 0x0030:  2025 5359 534c 4f47 2d35 2d4e 4f54 4943  .%SYSLOG-5-NOTIC
 0x0040:  453a 2061 6161 643a 2053 7562 5365 7373  E:.aaad:.SubSess
 0x0050:  696f 6e41 5554 4846 4149 4c20 7573 6572  ionAUTHFAIL.user
 0x0060:  3a20 7070 706f 6531 3633 3434 406d 6e20  :.pppoe16344@mn.
 0x0070:  2832 3429 2041 7574 6865 6e74 6963 6174  (24).Authenticat
 0x0080:  696f 6e20 6661 696c 7572 6520 5b43 6972  ion.failure.[Cir
 0x0090:  6375 6974 2068 616e 646c 653a 2031 2f34  cuit.handle:.1/4
 0x00a0:  3a35 3131 3a36 333a 3331 2f36 2f32 2f34  :511:63:31/6/2/4
 0x00b0:  3736 3631 5d0a   7661].

As you can see, the packet is much longer, but it doesn't exceed the 
maximum UDP packet size that can be processed by Graylog (8192). My guess 
is that logs coming from 10.50.255.111 are not RFC compatible and thus 
they're discarded by Graylog. How can I debug it / fix it? I didn't find 
any related messages in the Elasticsearch log (there were no errors related 
to parsing a message).
I deleted the default Input object and added a new RAW UDP Input object. It 
didn't fix the issue - logs from 10.50.255.111 are still not parsed.

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/bdf5da44-6854-4f54-b99c-421f5febe76f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Re: Extractor and processing messages

[Rayees Namathponnan]

Got it Thanks 



> On Feb 8, 2017, at 9:03 AM, Jochen Schalanda  wrote:
> 
> Hi Rayees,
> 
> On Wednesday, 8 February 2017 18:00:05 UTC+1, Rayees Namathponnan wrote:
> I am looking extractor configuration, there i am not seeing any way to define 
> the input,  without this all the messages comes to system will go trough the 
> extractor right ? I am missing something ?
> 
> Extractors are logically bound to inputs. You can create an (almost) 
> arbitrary number of inputs in Graylog for each special case.
> 
> Cheers,
> Jochen
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Graylog Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to graylog2+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/graylog2/ec04c1d9-e2be-4c97-9b42-4d0c7e9eb079%40googlegroups.com
>  
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/44A2EB03-756D-4DD9-AEEE-5B20F27E1F7D%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Re: Extractor and processing messages

[Jochen Schalanda]

Hi Rayees,

On Wednesday, 8 February 2017 18:00:05 UTC+1, Rayees Namathponnan wrote:
>
> I am looking extractor configuration, there i am not seeing any way to 
> define the input,  without this all the messages comes to system will go 
> trough the extractor right ? I am missing something ?
>

Extractors are logically bound to inputs. You can create an (almost) 
arbitrary number of inputs in Graylog for each special case.

Cheers,
Jochen

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/ec04c1d9-e2be-4c97-9b42-4d0c7e9eb079%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Re: Extractor and processing messages

[Rayees Namathponnan]

Regarding your second point 

I am looking extractor configuration, there i am not seeing any way to define 
the input,  without this all the messages comes to system will go trough the 
extractor right ? I am missing something ?




> On Feb 8, 2017, at 8:46 AM, Jochen Schalanda  wrote:
> 
> Hi Rayees,
> 
> On Wednesday, 8 February 2017 17:38:56 UTC+1, Rayees Namathponnan wrote:
> Suppose i have defined 10 extractors and if any messages comes to graylog 
> this go trough all the 10 extractors ?
> 
> This depends on your configuration and if the preconditions for these 
> extractors have been met, but it's possible that all 10 extractors have to 
> run for each message.
> 
> I am performing some test in graylog and see how graylog behave if i add more 
> extractor,  and want to check alert performance wrt to number of extractor 
> 
> Usually you would create multiple inputs for messages with vastly different 
> requirements to run extractors, so that not all extractors have to run for 
> all ingested messages.
> 
> Cheers,
> Jochen 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Graylog Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to graylog2+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/graylog2/01c71c04-d8bd-4283-ac8e-16f6926932a2%40googlegroups.com
>  
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/185278C5-78E1-4664-9461-595A7E93F329%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Extractor and processing messages

[Jochen Schalanda]

Hi Rayees,

On Wednesday, 8 February 2017 17:38:56 UTC+1, Rayees Namathponnan wrote:
>
> Suppose i have defined 10 extractors and if any messages comes to graylog 
> this go trough all the 10 extractors ?
>

This depends on your configuration and if the preconditions for these 
extractors have been met, but it's possible that all 10 extractors have to 
run for each message.

I am performing some test in graylog and see how graylog behave if i add 
> more extractor,  and want to check alert performance wrt to number of 
> extractor 
>

Usually you would create multiple inputs for messages with vastly different 
requirements to run extractors, so that not all extractors have to run for 
all ingested messages.

Cheers,
Jochen 

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/01c71c04-d8bd-4283-ac8e-16f6926932a2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Extractor and processing messages

[Rayees Namathponnan]

Hi All,

Suppose i have defined 10 extractors and if any messages comes to graylog this 
go trough all the 10 extractors ? 

I am performing some test in graylog and see how graylog behave if i add more 
extractor,  and want to check alert performance wrt to number of extractor 

Regards,
Rayees 


-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/E275D099-2979-413E-BAE2-1567669285D0%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Overwriting Timestamp field using Pipeline rules

[Al Reynolds]

That's what I get for typing it out...thank you for catching that! 
Unfortunately, even after correcting for the incorrect milliseconds value, 
it's still not replacing timestamp value. I sent the parsed date to a new 
field (in this case, "log_timestamp") to verify that the output data was in 
the correct format, which it is now, but it still won't replace the 
timestamp field.

Message sample with "log_timestamp" field:
WO_CS_RAS_CS_MESSAGE
2017-02-08 11:00:34,980 WARN  [Task 'ATLANTA-FS' FS timer.1] 
FriendshipTasksServiceImpl = Could not obtain task info for:  2c95ac8e-57e3-
91b2-0158-495b880b24e8REQUEST FAILED ==> STATUS CODE: 404, RESPONSE BODY:
WO_LogLevel
WARN
WO_Log_Source
RAS-CS
WO_Message
Could not obtain task info for:  2c95ac8e-57e3-91b2-0158-495b880b24e8REQUEST 
FAILED ==> STATUS CODE: 404, RESPONSE BODY:
WO_Process
Task 'ATLANTA-FS' FS timer.1
WO_SubProcess
FriendshipTasksServiceImpl
WO_Timestamp
2017-02-08 11:00:34,980
facility
filebeat
file
d:\centralserver\ras-server\log\ras_cs_WO-ATL-CS.log
input_type
log
log_timestamp
2017-02-08T11:00:34.980Z
message
2017-02-08 11:00:34,980 WARN  [Task 'ATLANTA-FS' FS timer.1] 
FriendshipTasksServiceImpl = Could not obtain task info for:  2c95ac8e-57e3-
91b2-0158-495b880b24e8REQUEST FAILED ==> STATUS CODE: 404, RESPONSE BODY:
name
WO-ATL-CS
offset
2372156
source
WO-ATL-CS
timestamp
2017-02-08T16:00:35.864Z
type
log

Corrected rule: 
rule "WO-CS-RAS" 
when 

contains(to_string($message.file),"centralserver\\ras-server\\log\\ras_cs_")
then
set_field("WO_Log_Source","RAS-CS");
let matches = grok(pattern: "%{WO_CS_RAS_CS_MESSAGE}", value: 
to_string($message.message));
set_fields(matches);
let date = parse_date(to_string($message.WO_Timestamp), "-MM-dd 
HH:mm:ss,SSS");
set_field("timestamp", date);
route_to_stream("WideOrbit Logs");
end

Thanks!

Cheers,
Al

On Wednesday, February 8, 2017 at 10:55:03 AM UTC-5, Jochen Schalanda wrote:
>
> Hi Al,
>
> On Wednesday, 8 February 2017 15:46:07 UTC+1, Al Reynolds wrote:
>>
>> WO_Timestamp
>> 2017-02-08 09:42:30,056
>>
>> Those messages are with the date parsing disabled. I'm attempting to 
>> replace "timestamp" with the "WO_Timestamp" field. 
>>
>
> The string in WO_Timestamp doesn't match the pattern "-MM-dd 
> HH:mm:ss,sss" used in parse_date(). See 
> http://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html
>  
> for details.
>
> Hint: 's' and 'S' are not the same thing.
>  
>
> Side note: The full_message field is empty on my filebeat inputs--is that 
>> expected behavior? 
>>
>
> Yes, that's expected.
>
> What would you expect to find in the (optional) full_message field?
>
> Cheers,
> Jochen
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/39dbaa3e-75d5-40c5-99f7-f4f2967ce134%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Overwriting Timestamp field using Pipeline rules

[Jochen Schalanda]

Hi Al,

On Wednesday, 8 February 2017 15:11:34 UTC+1, Al Reynolds wrote:
>
> I was under the impression that using the "parse_date" function would 
> create a Date object?
>

It does, 
see http://docs.graylog.org/en/2.1/pages/pipelines/functions.html#parse-date 
for reference. But your date pattern may be wrong 
(see 
http://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html 
for reference).

Please share some example messages, so that we can validate your rule.

 

> As for "$timestamp" instead of "timestamp", I was trying different 
> configurations, and thought that since the message field is referenced as 
> "$message" I would try that format. What does the "$" indicate? 
>

The $ character is simply part of the variable name containing the current 
message (which is "$message"). It doesn't have a special meaning.

Cheers,
Jochen

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/acd02ab0-564b-46cc-bab8-627170b05489%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Overwriting Timestamp field using Pipeline rules

[Al Reynolds]

Jochen,

Thanks for the reply! I'm guessing my problem is that the source field (in 
this case WO_Timestamp) is not a date object, as I'm not having any luck 
with your example either. I was under the impression that using the 
"parse_date" function would create a Date object? 

As for "$timestamp" instead of "timestamp", I was trying different 
configurations, and thought that since the message field is referenced as 
"$message" I would try that format. What does the "$" indicate? 

Thanks!

Cheers,
Al

On Wednesday, February 8, 2017 at 2:44:38 AM UTC-5, Jochen Schalanda wrote:
>
> Hi Al,
>
> the "timestamp" field has to be a Date object and not a string. 
> Additionally, the first parameter of your set_field() call seems odd 
> ("$timestamp" instead of "timestamp").
>
> This rule might work, although I haven't tested it:
>
> rule "WO-CS-RAS" 
> when 
> 
> contains(to_string($message.file),"centralserver\\ras-server\\log\\ras_cs_")
> then
> set_field("WO_Log_Source","RAS-CS");
> let matches = grok(pattern: "%{WO_CS_RAS_CS_MESSAGE}", value: 
> to_string($message.message));
> set_fields(matches);
> let date = parse_date(to_string($message.WO_Timestamp), "-MM-dd 
> HH:mm:ss,sss");
> set_field("timestamp", date);
> route_to_stream("WideOrbit Logs");
> end
>
>
> Cheers,
> Jochen
>
>
> On Tuesday, 7 February 2017 20:52:38 UTC+1, Al Reynolds wrote:
>>
>> Hello all,
>>
>> I'm attempting to switch our logging infrastructure from the ELK stack to 
>> Graylog, but I'm running into an issue with the pipeline rules and 
>> replacing the timestamp field. Rule below: 
>>
>> rule "WO-CS-RAS" 
>> when 
>> 
>> contains(to_string($message.file),"centralserver\\ras-server\\log\\ras_cs_")
>> then
>> set_field("WO_Log_Source","RAS-CS");
>> let matches = grok(pattern: "%{WO_CS_RAS_CS_MESSAGE}", value: 
>> to_string($message.message));
>> set_fields(matches);
>> let date = parse_date(to_string($message.WO_Timestamp), "-MM-dd 
>> HH:mm:ss,sss");
>> let new_date = format_date(date,"-MM-DD'T'HH:mm:ss.SSS");
>> set_field("$timestamp", new_date);
>> route_to_stream("WideOrbit Logs");
>> end
>>
>> I've tried without the date formatter as well--no luck there either. The 
>> rule will error out and not replace the timestamp field. Everything else 
>> works perfectly. Any suggestions as to where I might be going wrong? If I 
>> use an extractor I can replace the timestamp field, but I'd like to keep 
>> everything in one place if possible. 
>>
>> Thanks!
>>
>> Cheers,
>> Al
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/919e3853-8bb3-462c-b12c-908779a46bf1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Graylog 2.2.0-rc.1 lags while editing inputs

['Ha NN' via Graylog Users]

Hmm ok i installed 2.2 rc1 from the scratch and the problem seems to be 
gone. So i guess it has something to do with the upgrade from 2.1.3 to 2.2 
rc1.

Am Mittwoch, 8. Februar 2017 13:22:37 UTC+1 schrieb Ha NN:
>
> JVM:
>
> GRAYLOG_SERVER_JAVA_OPTS="-Xms4g -Xmx4g -XX:NewRatio=1 -server 
> -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled 
> -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC 
> -XX:-OmitStackTraceInFastThrow"
>
> Graylog only stuff which is used:
> elasticsearch_shards = 4
> elasticsearch_replicas = 0
> elasticsearch_index_prefix = graylog
> allow_leading_wildcard_searches = false
> allow_highlighting = false
> elasticsearch_cluster_name = graylog
> elasticsearch_analyzer = standard
> output_batch_size = 2000
> output_flush_interval = 1
> output_fault_count_threshold = 5
> output_fault_penalty_seconds = 30
> processbuffer_processors = 10
> outputbuffer_processors = 5
> processor_wait_strategy = blocking
> ring_size = 16384
> inputbuffer_ring_size = 16384
> inputbuffer_processors = 2
> inputbuffer_wait_strategy = blocking
> message_journal_enabled = true
> message_journal_dir = /var/lib/graylog-server/journal
> lb_recognition_period_seconds = 3
> mongodb_uri = mongodb://localhost/graylog2
> mongodb_max_connections = 1000
> mongodb_threads_allowed_to_block_multiplier = 5
> content_packs_dir = /usr/share/graylog-server/contentpacks
> content_packs_auto_load = grok-patterns.json
>
>
>
> Am Mittwoch, 8. Februar 2017 12:56:36 UTC+1 schrieb Jochen Schalanda:
>>
>> Hi,
>>
>> this is the start command for Elasticsearch, not Graylog.
>>
>> Please post the configuration of Graylog and the JVM settings for Graylog 
>> (see 
>> http://docs.graylog.org/en/2.1/pages/configuration/file_location.html 
>> for where to find them).
>>
>> Cheers,
>> Jochen
>>
>> On Wednesday, 8 February 2017 12:14:41 UTC+1, Ha NN wrote:
>>>
>>> It has 8 cores, 32GB ram
>>>
>>> JVM:
>>> /usr/bin/java -Xms18g -Xmx18g -Djava.awt.headless=true -XX:+UseParNewGC 
>>> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 
>>> -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError 
>>> -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true 
>>> -Des.path.home=/usr/share/elasticsearch -cp 
>>> /usr/share/elasticsearch/lib/elasticsearch-2.4.4.jar:/usr/share/elasticsearch/lib/*
>>>  
>>> org.elasticsearch.bootstrap.Elasticsearch start 
>>> -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid 
>>> -Des.default.path.home=/usr/share/elasticsearch 
>>> -Des.default.path.logs=/var/log/elasticsearch 
>>> -Des.default.path.data=/var/lib/elasticsearch 
>>> -Des.default.path.conf=/etc/elasticsearch
>>>
>>> Am Mittwoch, 8. Februar 2017 11:54:59 UTC+1 schrieb Jochen Schalanda:

 Hi,

 there are quite long GC pauses mentioned in your logs.

 What are the hardware specs of the machine(s) running Graylog and how 
 did you configure Graylog (also how are the JVM settings)?

 Cheers,
 Jochen

 On Wednesday, 8 February 2017 11:43:27 UTC+1, Ha NN wrote:
>
> Hi,
>
> i am testing Graylog 2.2.0-rc.1 with a gelf udp input plugin. I send 
> logs with rsyslog into it. I created some grok pattern extractors mostly 
> those ones ID=%{DATA:id} 
>
> Once created and you want to edit them it takes a very long time to 
> load the edit page and it seems graylog stops to process messages as you 
> will see the messages in/out counter at the top goes down to 0.
>
> I also noticed that for some messages the extractors does not apply 
> but they should.
>
> I have a one node setup. I use multiple indicies for different streams 
> (what a great feature!!!)
>
> You will find following in the log:
>
> 2017-02-08T11:11:59.376+01:00 WARN  [NodePingThread] Did not find meta 
> info of this node. Re-registering.
> 2017-02-08T11:12:02.265+01:00 INFO  [jvm] 
> [graylog-192b57c1-d456-4817-acff-d460547e7775] [gc][young][172980][17325] 
> duration [725ms], collections [1]/[2.8s], total [725ms]/[7m], memory 
> [1.7gb]->[1.1gb]/[3.8gb], all_pools {[young] 
> [853.1mb]->[204mb]/[1.6gb]}{[survivor] 
> [13.7mb]->[42.2mb]/[204.7mb]}{[old] 
> [943.2mb]->[943.7mb]/[2gb]}
> 2017-02-08T11:14:27.066+01:00 INFO  [ExtractorsResource] Updated 
> extractor <7e13da31-ed47-11e6-a18b-b083fec76da6> of type [grok] in input 
> <58949a5f6c6c8c6b200a1b3b>.
> 2017-02-08T11:16:28.641+01:00 WARN  [NodePingThread] Did not find meta 
> info of this node. Re-registering.
> 2017-02-08T11:17:15.605+01:00 INFO  [ExtractorsResource] Updated 
> extractor <3c954090-ea26-11e6-95c6-b083fec76da6> of type [grok] in input 
> <58949a5f6c6c8c6b200a1b3b>.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To 

[graylog2] Re: sidecar and nxlog collectors - query

[Peter Dudas]

*Hi,*

Why don't you create inputs and outputs per channel? For me this is logical 
as you can only select one channel per input.
One for Security, one for Application, one for System and so on...
We even have a different one for Network Policy Servers.

In the collector configuration you can have several input/output pairs for 
the same collector config and each can have its own config.
We have 6 inputs and 6 outputs in the collector for Domain controllers.
Security/Application/System - twice as they are sending the logs to 2 
different Graylog Cluster (2 different output, 1 for production cluster, 1 
for development).

Example Input config: AD query for security events.

**[System[(Level=1  or Level=2 or Level=3 or Level=4 or 
Level=0)]]*

Example Output Verbatim Config to tag and drop messages:
*#this configuration deletes the computer account logins (where 
TargetUserName end with $)*
* Exec if ($EventID == 4624 or $EventID == 4634  or $EventID == 4678) and 
($EventType == "AUDIT_SUCCESS")\*
*{   \*
*if $TargetUserName =~ /.\$/ {   \*
*$raw_event = "Time:" + $EventTime + ", EventID:" + 
$EventID + ", LogonType:" + $LogonType + ", User:" + $TargetDomainName + 
"\\" + $TargetUserName + ", IPAddr:" + $IPAddress + "\n"; \*
*file_write("C:\\Program Files 
(x86)\\nxlog\data\\security_drop.log",  $raw_event);\*
*drop(); \*
*}   \*
*}*
* Exec $tag = 'PCI-DSS';*
* Exec if $EventID == 1102 {$action = 'Log Clear';}*
* Exec if $EventID == 4608 {$action = 'Windows Start';}*
* Exec if $EventID == 4609 {$action = 'Windows Shutdown';}*
* Exec if $EventID == 4610 {$action = 'An authentication package was loaded 
by the Local Security Authority.';}*
* Exec if $EventID == 4611 {$action = 'A trusted logon process has 
registered with the Local Security Authority.';}*
* Exec if $EventID == 4612 {$action = 'Internal resources allocated for the 
queuing of security event messages have been exhausted, leading to the loss 
of some security event messages.';}*
* Exec if $EventID == 4614 {$action = 'A notification package was loaded by 
the Security Accounts Manager';}*
* Exec if $EventID == 4616 {$action = 'Server time out of synchronization 
with Domain Controller';}*
* Exec if $EventID == 4624 {$action = 'Successful Logon (on DC)';}*
* Exec if $EventID == 4625 {$action = 'Failed Logon attempts – All users';}*
* Exec if $EventID == 4634 {$action = 'logoff';}*
* Exec if $EventID == 4647 {$action = 'logoff initiated';}*
*and another 90 lines to go*

Adding configurations to outputs is pretty simple since the Verbatim 
configuration is possible. Just be careful to have a line feed at the end 
(empty line) otherwise the Graylog writes the  at the end of the 
last line and nxlog does not loads that output.

You can make  Snippets with special configuration too.
We use it to make special paths and add logfile rotation to the files left 
on the Nxlog endpoints:
*{{if .Windows}}*
*Moduledir %ROOT%\modules*
*CacheDir %ROOT%\data*
*Pidfile %ROOT%\data\nxlog.pid*
*SpoolDir %ROOT%\data*
*LogFile %ROOT%\data\nxlog.log*
*LogLevel INFO*

**
*Module  xm_fileop*
**

**
*Module  xm_fileop*
**
*When@daily*
*Execfile_cycle('%ROOT%\data\nxlog.log', 7);*
*Execfile_cycle('%ROOT%\data\security_drop.log', 7);*
* *
**
**
*  Path 57c587964158fb082fa0a4a8 => 58872007e8ad88038eb4ec3b*
**
**
*  Path 57c587d44158fb082fa0a4ea => 58872063e8ad88038eb4ec9d*
**
*{{end}}*


I presume these samples will give you some ideas how can you setup your 
system.
More info at this 
post: https://groups.google.com/forum/#!msg/graylog2/ppPiVCA1hMg/-JPyvjo_AgAJ

Peter Dudas
On Tuesday, 7 February 2017 16:16:48 UTC+1, Ľubo wrote:
>
> Hi all,
> i have questions to queries for nxlog collectors with sidecar for windows.
>
> In configure NXLog inputs there are Channl and query field. 
>
>
> Could you give more specific examples for querylist?
>
> \
>
> \
>
> *\
>
> *[System/Level=4]\
>
> *[Application/Level=2]\
>
> *[System/Level=3]\
>
> *\
>
> *\\
>
> 
>
>
> we need logs from all channel, but we do not need logs for examle Level=3, 
> or we do not need logs from specific source, do you have some experience 
> and could you share these querylist?
>
>
> there is below define nxlog snippets, too.
>
>
> I would like to see some more specific use case of nxlog snippets?
>
> Is it possible to use "Exec if ($Channel == 
> "Microsoft-Windows-WMI-Activity/Operational") drop();
>
> Exec if ($Channel == "Security") drop();"
>
>
> thanks for your help
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 

[graylog2] Re: Graylog 2.2.0-rc.1 lags while editing inputs

[Jochen Schalanda]

Hi,

this is the start command for Elasticsearch, not Graylog.

Please post the configuration of Graylog and the JVM settings for Graylog 
(see http://docs.graylog.org/en/2.1/pages/configuration/file_location.html 
for where to find them).

Cheers,
Jochen

On Wednesday, 8 February 2017 12:14:41 UTC+1, Ha NN wrote:
>
> It has 8 cores, 32GB ram
>
> JVM:
> /usr/bin/java -Xms18g -Xmx18g -Djava.awt.headless=true -XX:+UseParNewGC 
> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 
> -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError 
> -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true 
> -Des.path.home=/usr/share/elasticsearch -cp 
> /usr/share/elasticsearch/lib/elasticsearch-2.4.4.jar:/usr/share/elasticsearch/lib/*
>  
> org.elasticsearch.bootstrap.Elasticsearch start 
> -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid 
> -Des.default.path.home=/usr/share/elasticsearch 
> -Des.default.path.logs=/var/log/elasticsearch 
> -Des.default.path.data=/var/lib/elasticsearch 
> -Des.default.path.conf=/etc/elasticsearch
>
> Am Mittwoch, 8. Februar 2017 11:54:59 UTC+1 schrieb Jochen Schalanda:
>>
>> Hi,
>>
>> there are quite long GC pauses mentioned in your logs.
>>
>> What are the hardware specs of the machine(s) running Graylog and how did 
>> you configure Graylog (also how are the JVM settings)?
>>
>> Cheers,
>> Jochen
>>
>> On Wednesday, 8 February 2017 11:43:27 UTC+1, Ha NN wrote:
>>>
>>> Hi,
>>>
>>> i am testing Graylog 2.2.0-rc.1 with a gelf udp input plugin. I send 
>>> logs with rsyslog into it. I created some grok pattern extractors mostly 
>>> those ones ID=%{DATA:id} 
>>>
>>> Once created and you want to edit them it takes a very long time to load 
>>> the edit page and it seems graylog stops to process messages as you will 
>>> see the messages in/out counter at the top goes down to 0.
>>>
>>> I also noticed that for some messages the extractors does not apply but 
>>> they should.
>>>
>>> I have a one node setup. I use multiple indicies for different streams 
>>> (what a great feature!!!)
>>>
>>> You will find following in the log:
>>>
>>> 2017-02-08T11:11:59.376+01:00 WARN  [NodePingThread] Did not find meta 
>>> info of this node. Re-registering.
>>> 2017-02-08T11:12:02.265+01:00 INFO  [jvm] 
>>> [graylog-192b57c1-d456-4817-acff-d460547e7775] [gc][young][172980][17325] 
>>> duration [725ms], collections [1]/[2.8s], total [725ms]/[7m], memory 
>>> [1.7gb]->[1.1gb]/[3.8gb], all_pools {[young] 
>>> [853.1mb]->[204mb]/[1.6gb]}{[survivor] [13.7mb]->[42.2mb]/[204.7mb]}{[old] 
>>> [943.2mb]->[943.7mb]/[2gb]}
>>> 2017-02-08T11:14:27.066+01:00 INFO  [ExtractorsResource] Updated 
>>> extractor <7e13da31-ed47-11e6-a18b-b083fec76da6> of type [grok] in input 
>>> <58949a5f6c6c8c6b200a1b3b>.
>>> 2017-02-08T11:16:28.641+01:00 WARN  [NodePingThread] Did not find meta 
>>> info of this node. Re-registering.
>>> 2017-02-08T11:17:15.605+01:00 INFO  [ExtractorsResource] Updated 
>>> extractor <3c954090-ea26-11e6-95c6-b083fec76da6> of type [grok] in input 
>>> <58949a5f6c6c8c6b200a1b3b>.
>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/a4e61733-a6f7-4fec-b4af-3888543c4f0e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Graylog 2.2.0-rc.1 lags while editing inputs

['Ha NN' via Graylog Users]

It has 8 cores, 32GB ram

JVM:
/usr/bin/java -Xms18g -Xmx18g -Djava.awt.headless=true -XX:+UseParNewGC 
-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 
-XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError 
-XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true 
-Des.path.home=/usr/share/elasticsearch -cp 
/usr/share/elasticsearch/lib/elasticsearch-2.4.4.jar:/usr/share/elasticsearch/lib/*
 
org.elasticsearch.bootstrap.Elasticsearch start 
-Des.pidfile=/var/run/elasticsearch/elasticsearch.pid 
-Des.default.path.home=/usr/share/elasticsearch 
-Des.default.path.logs=/var/log/elasticsearch 
-Des.default.path.data=/var/lib/elasticsearch 
-Des.default.path.conf=/etc/elasticsearch

Am Mittwoch, 8. Februar 2017 11:54:59 UTC+1 schrieb Jochen Schalanda:
>
> Hi,
>
> there are quite long GC pauses mentioned in your logs.
>
> What are the hardware specs of the machine(s) running Graylog and how did 
> you configure Graylog (also how are the JVM settings)?
>
> Cheers,
> Jochen
>
> On Wednesday, 8 February 2017 11:43:27 UTC+1, Ha NN wrote:
>>
>> Hi,
>>
>> i am testing Graylog 2.2.0-rc.1 with a gelf udp input plugin. I send logs 
>> with rsyslog into it. I created some grok pattern extractors mostly those 
>> ones ID=%{DATA:id} 
>>
>> Once created and you want to edit them it takes a very long time to load 
>> the edit page and it seems graylog stops to process messages as you will 
>> see the messages in/out counter at the top goes down to 0.
>>
>> I also noticed that for some messages the extractors does not apply but 
>> they should.
>>
>> I have a one node setup. I use multiple indicies for different streams 
>> (what a great feature!!!)
>>
>> You will find following in the log:
>>
>> 2017-02-08T11:11:59.376+01:00 WARN  [NodePingThread] Did not find meta 
>> info of this node. Re-registering.
>> 2017-02-08T11:12:02.265+01:00 INFO  [jvm] 
>> [graylog-192b57c1-d456-4817-acff-d460547e7775] [gc][young][172980][17325] 
>> duration [725ms], collections [1]/[2.8s], total [725ms]/[7m], memory 
>> [1.7gb]->[1.1gb]/[3.8gb], all_pools {[young] 
>> [853.1mb]->[204mb]/[1.6gb]}{[survivor] [13.7mb]->[42.2mb]/[204.7mb]}{[old] 
>> [943.2mb]->[943.7mb]/[2gb]}
>> 2017-02-08T11:14:27.066+01:00 INFO  [ExtractorsResource] Updated 
>> extractor <7e13da31-ed47-11e6-a18b-b083fec76da6> of type [grok] in input 
>> <58949a5f6c6c8c6b200a1b3b>.
>> 2017-02-08T11:16:28.641+01:00 WARN  [NodePingThread] Did not find meta 
>> info of this node. Re-registering.
>> 2017-02-08T11:17:15.605+01:00 INFO  [ExtractorsResource] Updated 
>> extractor <3c954090-ea26-11e6-95c6-b083fec76da6> of type [grok] in input 
>> <58949a5f6c6c8c6b200a1b3b>.
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/c967bdd8-d53a-4678-834f-fd98ae00b9e4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Graylog 2.2.0-rc.1 lags while editing inputs

[Jochen Schalanda]

Hi,

there are quite long GC pauses mentioned in your logs.

What are the hardware specs of the machine(s) running Graylog and how did 
you configure Graylog (also how are the JVM settings)?

Cheers,
Jochen

On Wednesday, 8 February 2017 11:43:27 UTC+1, Ha NN wrote:
>
> Hi,
>
> i am testing Graylog 2.2.0-rc.1 with a gelf udp input plugin. I send logs 
> with rsyslog into it. I created some grok pattern extractors mostly those 
> ones ID=%{DATA:id} 
>
> Once created and you want to edit them it takes a very long time to load 
> the edit page and it seems graylog stops to process messages as you will 
> see the messages in/out counter at the top goes down to 0.
>
> I also noticed that for some messages the extractors does not apply but 
> they should.
>
> I have a one node setup. I use multiple indicies for different streams 
> (what a great feature!!!)
>
> You will find following in the log:
>
> 2017-02-08T11:11:59.376+01:00 WARN  [NodePingThread] Did not find meta 
> info of this node. Re-registering.
> 2017-02-08T11:12:02.265+01:00 INFO  [jvm] 
> [graylog-192b57c1-d456-4817-acff-d460547e7775] [gc][young][172980][17325] 
> duration [725ms], collections [1]/[2.8s], total [725ms]/[7m], memory 
> [1.7gb]->[1.1gb]/[3.8gb], all_pools {[young] 
> [853.1mb]->[204mb]/[1.6gb]}{[survivor] [13.7mb]->[42.2mb]/[204.7mb]}{[old] 
> [943.2mb]->[943.7mb]/[2gb]}
> 2017-02-08T11:14:27.066+01:00 INFO  [ExtractorsResource] Updated extractor 
> <7e13da31-ed47-11e6-a18b-b083fec76da6> of type [grok] in input 
> <58949a5f6c6c8c6b200a1b3b>.
> 2017-02-08T11:16:28.641+01:00 WARN  [NodePingThread] Did not find meta 
> info of this node. Re-registering.
> 2017-02-08T11:17:15.605+01:00 INFO  [ExtractorsResource] Updated extractor 
> <3c954090-ea26-11e6-95c6-b083fec76da6> of type [grok] in input 
> <58949a5f6c6c8c6b200a1b3b>.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/9bf9b698-1f06-48fc-adcb-642cf4ad7198%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Graylog 2.2.0-rc.1 lags while editing inputs

['Ha NN' via Graylog Users]

Hi,

i am testing Graylog 2.2.0-rc.1 with a gelf udp input plugin. I send logs 
with rsyslog into it. I created some grok pattern extractors mostly those 
ones ID=%{DATA:id} 

Once created and you want to edit them it takes a very long time to load 
the edit page and it seems graylog stops to process messages as you will 
see the messages in/out counter at the top goes down to 0.

I also noticed that for some messages the extractors does not apply but 
they should.

I have a one node setup. I use multiple indicies for different streams 
(what a great feature!!!)

You will find following in the log:

2017-02-08T11:11:59.376+01:00 WARN  [NodePingThread] Did not find meta info 
of this node. Re-registering.
2017-02-08T11:12:02.265+01:00 INFO  [jvm] 
[graylog-192b57c1-d456-4817-acff-d460547e7775] [gc][young][172980][17325] 
duration [725ms], collections [1]/[2.8s], total [725ms]/[7m], memory 
[1.7gb]->[1.1gb]/[3.8gb], all_pools {[young] 
[853.1mb]->[204mb]/[1.6gb]}{[survivor] [13.7mb]->[42.2mb]/[204.7mb]}{[old] 
[943.2mb]->[943.7mb]/[2gb]}
2017-02-08T11:14:27.066+01:00 INFO  [ExtractorsResource] Updated extractor 
<7e13da31-ed47-11e6-a18b-b083fec76da6> of type [grok] in input 
<58949a5f6c6c8c6b200a1b3b>.
2017-02-08T11:16:28.641+01:00 WARN  [NodePingThread] Did not find meta info 
of this node. Re-registering.
2017-02-08T11:17:15.605+01:00 INFO  [ExtractorsResource] Updated extractor 
<3c954090-ea26-11e6-95c6-b083fec76da6> of type [grok] in input 
<58949a5f6c6c8c6b200a1b3b>.

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/d68261fd-f5dc-490b-9720-d637f0ab60de%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.