[graylog2] Re: Overwriting Timestamp field using Pipeline rules
Hi Al,
you might want to try to use your pattern with lower case 'y' for the year
component of the date pattern.
Cheers,
Jochen
On Wednesday, 8 February 2017 21:09:19 UTC+1, Al Reynolds wrote:
>
> I've noticed another error. The timestamp field is being replaced
> correctly, but the "gl2_processing_error" field is showing the following
> error (on all messages):
> For rule 'WO-CS-RAS': In call to function 'parse_date' at 8:15 an
> exception was thrown: Invalid format: "2017-02-08 15:05:59,170" is
> malformed at "17-02-08 15:05:59,170"
>
>
> It doesn't seem to have any adverse effects, but I'm curious as to what
> might be causing it?
>
> On Wednesday, February 8, 2017 at 1:56:17 PM UTC-5, Al Reynolds wrote:
>>
>> Figured it out--parse_date needed the timestamp . New rule looks like
>> this:
>> rule "WO-CS-RAS"
>> when
>>
>> contains(to_string($message.file),"centralserver\\ras-server\\log\\ras_cs_")
>> then
>> set_field("WO_Log_Source","RAS-CS");
>> let matches = grok(pattern: "%{WO_CS_RAS_CS_MESSAGE}", value:
>> to_string($message.message));
>> set_fields(matches);
>> let date = parse_date(to_string($message.WO_Timestamp), "-MM-dd
>> HH:mm:ss,SSS", "EST");
>> set_field("timestamp", date);
>> route_to_stream("WideOrbit Logs");
>> end
>>
>> I was under the impression that the timezone was optional?
>>
>> Thanks for all your help with this Jochen--it's greatly appreciated!
>>
>> Cheers,
>> Al
>>
>> On Wednesday, February 8, 2017 at 11:05:22 AM UTC-5, Al Reynolds wrote:
>>>
>>> That's what I get for typing it out...thank you for catching that!
>>> Unfortunately, even after correcting for the incorrect milliseconds value,
>>> it's still not replacing timestamp value. I sent the parsed date to a new
>>> field (in this case, "log_timestamp") to verify that the output data was in
>>> the correct format, which it is now, but it still won't replace the
>>> timestamp field.
>>>
>>> Message sample with "log_timestamp" field:
>>> WO_CS_RAS_CS_MESSAGE
>>> 2017-02-08 11:00:34,980 WARN [Task 'ATLANTA-FS' FS timer.1]
>>> FriendshipTasksServiceImpl = Could not obtain task info for:
>>> 2c95ac8e-57e3-91b2-0158-495b880b24e8REQUEST FAILED ==> STATUS CODE: 404,
>>> RESPONSE BODY:
>>> WO_LogLevel
>>> WARN
>>> WO_Log_Source
>>> RAS-CS
>>> WO_Message
>>> Could not obtain task info for: 2c95ac8e-57e3-91b2-0158-
>>> 495b880b24e8REQUEST FAILED ==> STATUS CODE: 404, RESPONSE BODY:
>>> WO_Process
>>> Task 'ATLANTA-FS' FS timer.1
>>> WO_SubProcess
>>> FriendshipTasksServiceImpl
>>> WO_Timestamp
>>> 2017-02-08 11:00:34,980
>>> facility
>>> filebeat
>>> file
>>> d:\centralserver\ras-server\log\ras_cs_WO-ATL-CS.log
>>> input_type
>>> log
>>> log_timestamp
>>> 2017-02-08T11:00:34.980Z
>>> message
>>> 2017-02-08 11:00:34,980 WARN [Task 'ATLANTA-FS' FS timer.1]
>>> FriendshipTasksServiceImpl = Could not obtain task info for:
>>> 2c95ac8e-57e3-91b2-0158-495b880b24e8REQUEST FAILED ==> STATUS CODE: 404,
>>> RESPONSE BODY:
>>> name
>>> WO-ATL-CS
>>> offset
>>> 2372156
>>> source
>>> WO-ATL-CS
>>> timestamp
>>> 2017-02-08T16:00:35.864Z
>>> type
>>> log
>>>
>>> Corrected rule:
>>> rule "WO-CS-RAS"
>>> when
>>>
>>> contains(to_string($message.file),"centralserver\\ras-server\\log\\ras_cs_")
>>> then
>>> set_field("WO_Log_Source","RAS-CS");
>>> let matches = grok(pattern: "%{WO_CS_RAS_CS_MESSAGE}", value:
>>> to_string($message.message));
>>> set_fields(matches);
>>> let date = parse_date(to_string($message.WO_Timestamp), "-MM-dd
>>> HH:mm:ss,SSS");
>>> set_field("timestamp", date);
>>> route_to_stream("WideOrbit Logs");
>>> end
>>>
>>> Thanks!
>>>
>>> Cheers,
>>> Al
>>>
>>> On Wednesday, February 8, 2017 at 10:55:03 AM UTC-5, Jochen Schalanda
>>> wrote:
Hi Al,
On Wednesday, 8 February 2017 15:46:07 UTC+1, Al Reynolds wrote:
>
> WO_Timestamp
> 2017-02-08 09:42:30,056
>
> Those messages are with the date parsing disabled. I'm attempting to
> replace "timestamp" with the "WO_Timestamp" field.
>
The string in WO_Timestamp doesn't match the pattern "-MM-dd
HH:mm:ss,sss" used in parse_date(). See
http://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html
for details.
Hint: 's' and 'S' are not the same thing.
Side note: The full_message field is empty on my filebeat inputs--is
> that expected behavior?
>
Yes, that's expected.
What would you expect to find in the (optional) full_message field?
Cheers,
Jochen
>>>
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/561bf48b-bfdc-4ee6-b0a0-f7e706dc1e5c%40googlegroups.com.
[graylog2] Re: Overwriting Timestamp field using Pipeline rules
I've noticed another error. The timestamp field is being replaced
correctly, but the "gl2_processing_error" field is showing the following
error (on all messages):
For rule 'WO-CS-RAS': In call to function 'parse_date' at 8:15 an exception
was thrown: Invalid format: "2017-02-08 15:05:59,170" is malformed at "17-02-08
15:05:59,170"
It doesn't seem to have any adverse effects, but I'm curious as to what
might be causing it?
On Wednesday, February 8, 2017 at 1:56:17 PM UTC-5, Al Reynolds wrote:
>
> Figured it out--parse_date needed the timestamp . New rule looks like this:
> rule "WO-CS-RAS"
> when
>
> contains(to_string($message.file),"centralserver\\ras-server\\log\\ras_cs_")
> then
> set_field("WO_Log_Source","RAS-CS");
> let matches = grok(pattern: "%{WO_CS_RAS_CS_MESSAGE}", value:
> to_string($message.message));
> set_fields(matches);
> let date = parse_date(to_string($message.WO_Timestamp), "-MM-dd
> HH:mm:ss,SSS", "EST");
> set_field("timestamp", date);
> route_to_stream("WideOrbit Logs");
> end
>
> I was under the impression that the timezone was optional?
>
> Thanks for all your help with this Jochen--it's greatly appreciated!
>
> Cheers,
> Al
>
> On Wednesday, February 8, 2017 at 11:05:22 AM UTC-5, Al Reynolds wrote:
>>
>> That's what I get for typing it out...thank you for catching that!
>> Unfortunately, even after correcting for the incorrect milliseconds value,
>> it's still not replacing timestamp value. I sent the parsed date to a new
>> field (in this case, "log_timestamp") to verify that the output data was in
>> the correct format, which it is now, but it still won't replace the
>> timestamp field.
>>
>> Message sample with "log_timestamp" field:
>> WO_CS_RAS_CS_MESSAGE
>> 2017-02-08 11:00:34,980 WARN [Task 'ATLANTA-FS' FS timer.1]
>> FriendshipTasksServiceImpl = Could not obtain task info for:
>> 2c95ac8e-57e3-91b2-0158-495b880b24e8REQUEST FAILED ==> STATUS CODE: 404,
>> RESPONSE BODY:
>> WO_LogLevel
>> WARN
>> WO_Log_Source
>> RAS-CS
>> WO_Message
>> Could not obtain task info for: 2c95ac8e-57e3-91b2-0158-
>> 495b880b24e8REQUEST FAILED ==> STATUS CODE: 404, RESPONSE BODY:
>> WO_Process
>> Task 'ATLANTA-FS' FS timer.1
>> WO_SubProcess
>> FriendshipTasksServiceImpl
>> WO_Timestamp
>> 2017-02-08 11:00:34,980
>> facility
>> filebeat
>> file
>> d:\centralserver\ras-server\log\ras_cs_WO-ATL-CS.log
>> input_type
>> log
>> log_timestamp
>> 2017-02-08T11:00:34.980Z
>> message
>> 2017-02-08 11:00:34,980 WARN [Task 'ATLANTA-FS' FS timer.1]
>> FriendshipTasksServiceImpl = Could not obtain task info for:
>> 2c95ac8e-57e3-91b2-0158-495b880b24e8REQUEST FAILED ==> STATUS CODE: 404,
>> RESPONSE BODY:
>> name
>> WO-ATL-CS
>> offset
>> 2372156
>> source
>> WO-ATL-CS
>> timestamp
>> 2017-02-08T16:00:35.864Z
>> type
>> log
>>
>> Corrected rule:
>> rule "WO-CS-RAS"
>> when
>>
>> contains(to_string($message.file),"centralserver\\ras-server\\log\\ras_cs_")
>> then
>> set_field("WO_Log_Source","RAS-CS");
>> let matches = grok(pattern: "%{WO_CS_RAS_CS_MESSAGE}", value:
>> to_string($message.message));
>> set_fields(matches);
>> let date = parse_date(to_string($message.WO_Timestamp), "-MM-dd
>> HH:mm:ss,SSS");
>> set_field("timestamp", date);
>> route_to_stream("WideOrbit Logs");
>> end
>>
>> Thanks!
>>
>> Cheers,
>> Al
>>
>> On Wednesday, February 8, 2017 at 10:55:03 AM UTC-5, Jochen Schalanda
>> wrote:
>>>
>>> Hi Al,
>>>
>>> On Wednesday, 8 February 2017 15:46:07 UTC+1, Al Reynolds wrote:
WO_Timestamp
2017-02-08 09:42:30,056
Those messages are with the date parsing disabled. I'm attempting to
replace "timestamp" with the "WO_Timestamp" field.
>>>
>>> The string in WO_Timestamp doesn't match the pattern "-MM-dd
>>> HH:mm:ss,sss" used in parse_date(). See
>>> http://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html
>>>
>>> for details.
>>>
>>> Hint: 's' and 'S' are not the same thing.
>>>
>>>
>>> Side note: The full_message field is empty on my filebeat inputs--is
that expected behavior?
>>>
>>> Yes, that's expected.
>>>
>>> What would you expect to find in the (optional) full_message field?
>>>
>>> Cheers,
>>> Jochen
>>>
>>
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/34c557b5-8552-405b-8892-04c8eb1ad2b1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Overwriting Timestamp field using Pipeline rules
Figured it out--parse_date needed the timestamp . New rule looks like this:
rule "WO-CS-RAS"
when
contains(to_string($message.file),"centralserver\\ras-server\\log\\ras_cs_")
then
set_field("WO_Log_Source","RAS-CS");
let matches = grok(pattern: "%{WO_CS_RAS_CS_MESSAGE}", value:
to_string($message.message));
set_fields(matches);
let date = parse_date(to_string($message.WO_Timestamp), "-MM-dd
HH:mm:ss,SSS", "EST");
set_field("timestamp", date);
route_to_stream("WideOrbit Logs");
end
I was under the impression that the timezone was optional?
Thanks for all your help with this Jochen--it's greatly appreciated!
Cheers,
Al
On Wednesday, February 8, 2017 at 11:05:22 AM UTC-5, Al Reynolds wrote:
>
> That's what I get for typing it out...thank you for catching that!
> Unfortunately, even after correcting for the incorrect milliseconds value,
> it's still not replacing timestamp value. I sent the parsed date to a new
> field (in this case, "log_timestamp") to verify that the output data was in
> the correct format, which it is now, but it still won't replace the
> timestamp field.
>
> Message sample with "log_timestamp" field:
> WO_CS_RAS_CS_MESSAGE
> 2017-02-08 11:00:34,980 WARN [Task 'ATLANTA-FS' FS timer.1]
> FriendshipTasksServiceImpl = Could not obtain task info for:
> 2c95ac8e-57e3-91b2-0158-495b880b24e8REQUEST FAILED ==> STATUS CODE: 404,
> RESPONSE BODY:
> WO_LogLevel
> WARN
> WO_Log_Source
> RAS-CS
> WO_Message
> Could not obtain task info for: 2c95ac8e-57e3-91b2-0158-
> 495b880b24e8REQUEST FAILED ==> STATUS CODE: 404, RESPONSE BODY:
> WO_Process
> Task 'ATLANTA-FS' FS timer.1
> WO_SubProcess
> FriendshipTasksServiceImpl
> WO_Timestamp
> 2017-02-08 11:00:34,980
> facility
> filebeat
> file
> d:\centralserver\ras-server\log\ras_cs_WO-ATL-CS.log
> input_type
> log
> log_timestamp
> 2017-02-08T11:00:34.980Z
> message
> 2017-02-08 11:00:34,980 WARN [Task 'ATLANTA-FS' FS timer.1]
> FriendshipTasksServiceImpl = Could not obtain task info for:
> 2c95ac8e-57e3-91b2-0158-495b880b24e8REQUEST FAILED ==> STATUS CODE: 404,
> RESPONSE BODY:
> name
> WO-ATL-CS
> offset
> 2372156
> source
> WO-ATL-CS
> timestamp
> 2017-02-08T16:00:35.864Z
> type
> log
>
> Corrected rule:
> rule "WO-CS-RAS"
> when
>
> contains(to_string($message.file),"centralserver\\ras-server\\log\\ras_cs_")
> then
> set_field("WO_Log_Source","RAS-CS");
> let matches = grok(pattern: "%{WO_CS_RAS_CS_MESSAGE}", value:
> to_string($message.message));
> set_fields(matches);
> let date = parse_date(to_string($message.WO_Timestamp), "-MM-dd
> HH:mm:ss,SSS");
> set_field("timestamp", date);
> route_to_stream("WideOrbit Logs");
> end
>
> Thanks!
>
> Cheers,
> Al
>
> On Wednesday, February 8, 2017 at 10:55:03 AM UTC-5, Jochen Schalanda
> wrote:
>>
>> Hi Al,
>>
>> On Wednesday, 8 February 2017 15:46:07 UTC+1, Al Reynolds wrote:
>>>
>>> WO_Timestamp
>>> 2017-02-08 09:42:30,056
>>>
>>> Those messages are with the date parsing disabled. I'm attempting to
>>> replace "timestamp" with the "WO_Timestamp" field.
>>>
>>
>> The string in WO_Timestamp doesn't match the pattern "-MM-dd
>> HH:mm:ss,sss" used in parse_date(). See
>> http://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html
>>
>> for details.
>>
>> Hint: 's' and 'S' are not the same thing.
>>
>>
>> Side note: The full_message field is empty on my filebeat inputs--is that
>>> expected behavior?
>>>
>>
>> Yes, that's expected.
>>
>> What would you expect to find in the (optional) full_message field?
>>
>> Cheers,
>> Jochen
>>
>
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/a3b908a8-4993-4107-87f8-55445605cc66%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Overwriting Timestamp field using Pipeline rules
That's what I get for typing it out...thank you for catching that!
Unfortunately, even after correcting for the incorrect milliseconds value,
it's still not replacing timestamp value. I sent the parsed date to a new
field (in this case, "log_timestamp") to verify that the output data was in
the correct format, which it is now, but it still won't replace the
timestamp field.
Message sample with "log_timestamp" field:
WO_CS_RAS_CS_MESSAGE
2017-02-08 11:00:34,980 WARN [Task 'ATLANTA-FS' FS timer.1]
FriendshipTasksServiceImpl = Could not obtain task info for: 2c95ac8e-57e3-
91b2-0158-495b880b24e8REQUEST FAILED ==> STATUS CODE: 404, RESPONSE BODY:
WO_LogLevel
WARN
WO_Log_Source
RAS-CS
WO_Message
Could not obtain task info for: 2c95ac8e-57e3-91b2-0158-495b880b24e8REQUEST
FAILED ==> STATUS CODE: 404, RESPONSE BODY:
WO_Process
Task 'ATLANTA-FS' FS timer.1
WO_SubProcess
FriendshipTasksServiceImpl
WO_Timestamp
2017-02-08 11:00:34,980
facility
filebeat
file
d:\centralserver\ras-server\log\ras_cs_WO-ATL-CS.log
input_type
log
log_timestamp
2017-02-08T11:00:34.980Z
message
2017-02-08 11:00:34,980 WARN [Task 'ATLANTA-FS' FS timer.1]
FriendshipTasksServiceImpl = Could not obtain task info for: 2c95ac8e-57e3-
91b2-0158-495b880b24e8REQUEST FAILED ==> STATUS CODE: 404, RESPONSE BODY:
name
WO-ATL-CS
offset
2372156
source
WO-ATL-CS
timestamp
2017-02-08T16:00:35.864Z
type
log
Corrected rule:
rule "WO-CS-RAS"
when
contains(to_string($message.file),"centralserver\\ras-server\\log\\ras_cs_")
then
set_field("WO_Log_Source","RAS-CS");
let matches = grok(pattern: "%{WO_CS_RAS_CS_MESSAGE}", value:
to_string($message.message));
set_fields(matches);
let date = parse_date(to_string($message.WO_Timestamp), "-MM-dd
HH:mm:ss,SSS");
set_field("timestamp", date);
route_to_stream("WideOrbit Logs");
end
Thanks!
Cheers,
Al
On Wednesday, February 8, 2017 at 10:55:03 AM UTC-5, Jochen Schalanda wrote:
>
> Hi Al,
>
> On Wednesday, 8 February 2017 15:46:07 UTC+1, Al Reynolds wrote:
>>
>> WO_Timestamp
>> 2017-02-08 09:42:30,056
>>
>> Those messages are with the date parsing disabled. I'm attempting to
>> replace "timestamp" with the "WO_Timestamp" field.
>>
>
> The string in WO_Timestamp doesn't match the pattern "-MM-dd
> HH:mm:ss,sss" used in parse_date(). See
> http://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html
>
> for details.
>
> Hint: 's' and 'S' are not the same thing.
>
>
> Side note: The full_message field is empty on my filebeat inputs--is that
>> expected behavior?
>>
>
> Yes, that's expected.
>
> What would you expect to find in the (optional) full_message field?
>
> Cheers,
> Jochen
>
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/39dbaa3e-75d5-40c5-99f7-f4f2967ce134%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Overwriting Timestamp field using Pipeline rules
Hi Al, On Wednesday, 8 February 2017 15:46:07 UTC+1, Al Reynolds wrote: > > WO_Timestamp > 2017-02-08 09:42:30,056 > > Those messages are with the date parsing disabled. I'm attempting to > replace "timestamp" with the "WO_Timestamp" field. > The string in WO_Timestamp doesn't match the pattern "-MM-dd HH:mm:ss,sss" used in parse_date(). See http://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html for details. Hint: 's' and 'S' are not the same thing. Side note: The full_message field is empty on my filebeat inputs--is that > expected behavior? > Yes, that's expected. What would you expect to find in the (optional) full_message field? Cheers, Jochen -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/8e3d890e-8c2a-473e-b9c4-8d17f65ac47e%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Overwriting Timestamp field using Pipeline rules
Hi Al, On Wednesday, 8 February 2017 15:11:34 UTC+1, Al Reynolds wrote: > > I was under the impression that using the "parse_date" function would > create a Date object? > It does, see http://docs.graylog.org/en/2.1/pages/pipelines/functions.html#parse-date for reference. But your date pattern may be wrong (see http://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html for reference). Please share some example messages, so that we can validate your rule. > As for "$timestamp" instead of "timestamp", I was trying different > configurations, and thought that since the message field is referenced as > "$message" I would try that format. What does the "$" indicate? > The $ character is simply part of the variable name containing the current message (which is "$message"). It doesn't have a special meaning. Cheers, Jochen -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/acd02ab0-564b-46cc-bab8-627170b05489%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Overwriting Timestamp field using Pipeline rules
Jochen,
Thanks for the reply! I'm guessing my problem is that the source field (in
this case WO_Timestamp) is not a date object, as I'm not having any luck
with your example either. I was under the impression that using the
"parse_date" function would create a Date object?
As for "$timestamp" instead of "timestamp", I was trying different
configurations, and thought that since the message field is referenced as
"$message" I would try that format. What does the "$" indicate?
Thanks!
Cheers,
Al
On Wednesday, February 8, 2017 at 2:44:38 AM UTC-5, Jochen Schalanda wrote:
>
> Hi Al,
>
> the "timestamp" field has to be a Date object and not a string.
> Additionally, the first parameter of your set_field() call seems odd
> ("$timestamp" instead of "timestamp").
>
> This rule might work, although I haven't tested it:
>
> rule "WO-CS-RAS"
> when
>
> contains(to_string($message.file),"centralserver\\ras-server\\log\\ras_cs_")
> then
> set_field("WO_Log_Source","RAS-CS");
> let matches = grok(pattern: "%{WO_CS_RAS_CS_MESSAGE}", value:
> to_string($message.message));
> set_fields(matches);
> let date = parse_date(to_string($message.WO_Timestamp), "-MM-dd
> HH:mm:ss,sss");
> set_field("timestamp", date);
> route_to_stream("WideOrbit Logs");
> end
>
>
> Cheers,
> Jochen
>
>
> On Tuesday, 7 February 2017 20:52:38 UTC+1, Al Reynolds wrote:
>>
>> Hello all,
>>
>> I'm attempting to switch our logging infrastructure from the ELK stack to
>> Graylog, but I'm running into an issue with the pipeline rules and
>> replacing the timestamp field. Rule below:
>>
>> rule "WO-CS-RAS"
>> when
>>
>> contains(to_string($message.file),"centralserver\\ras-server\\log\\ras_cs_")
>> then
>> set_field("WO_Log_Source","RAS-CS");
>> let matches = grok(pattern: "%{WO_CS_RAS_CS_MESSAGE}", value:
>> to_string($message.message));
>> set_fields(matches);
>> let date = parse_date(to_string($message.WO_Timestamp), "-MM-dd
>> HH:mm:ss,sss");
>> let new_date = format_date(date,"-MM-DD'T'HH:mm:ss.SSS");
>> set_field("$timestamp", new_date);
>> route_to_stream("WideOrbit Logs");
>> end
>>
>> I've tried without the date formatter as well--no luck there either. The
>> rule will error out and not replace the timestamp field. Everything else
>> works perfectly. Any suggestions as to where I might be going wrong? If I
>> use an extractor I can replace the timestamp field, but I'd like to keep
>> everything in one place if possible.
>>
>> Thanks!
>>
>> Cheers,
>> Al
>>
>
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/919e3853-8bb3-462c-b12c-908779a46bf1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Overwriting Timestamp field using Pipeline rules
Hi Al,
the "timestamp" field has to be a Date object and not a string.
Additionally, the first parameter of your set_field() call seems odd
("$timestamp" instead of "timestamp").
This rule might work, although I haven't tested it:
rule "WO-CS-RAS"
when
contains(to_string($message.file),"centralserver\\ras-server\\log\\ras_cs_")
then
set_field("WO_Log_Source","RAS-CS");
let matches = grok(pattern: "%{WO_CS_RAS_CS_MESSAGE}", value:
to_string($message.message));
set_fields(matches);
let date = parse_date(to_string($message.WO_Timestamp), "-MM-dd
HH:mm:ss,sss");
set_field("timestamp", date);
route_to_stream("WideOrbit Logs");
end
Cheers,
Jochen
On Tuesday, 7 February 2017 20:52:38 UTC+1, Al Reynolds wrote:
>
> Hello all,
>
> I'm attempting to switch our logging infrastructure from the ELK stack to
> Graylog, but I'm running into an issue with the pipeline rules and
> replacing the timestamp field. Rule below:
>
> rule "WO-CS-RAS"
> when
>
> contains(to_string($message.file),"centralserver\\ras-server\\log\\ras_cs_")
> then
> set_field("WO_Log_Source","RAS-CS");
> let matches = grok(pattern: "%{WO_CS_RAS_CS_MESSAGE}", value:
> to_string($message.message));
> set_fields(matches);
> let date = parse_date(to_string($message.WO_Timestamp), "-MM-dd
> HH:mm:ss,sss");
> let new_date = format_date(date,"-MM-DD'T'HH:mm:ss.SSS");
> set_field("$timestamp", new_date);
> route_to_stream("WideOrbit Logs");
> end
>
> I've tried without the date formatter as well--no luck there either. The
> rule will error out and not replace the timestamp field. Everything else
> works perfectly. Any suggestions as to where I might be going wrong? If I
> use an extractor I can replace the timestamp field, but I'd like to keep
> everything in one place if possible.
>
> Thanks!
>
> Cheers,
> Al
>
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/b1ee4811-c22a-4529-8544-f23c5411bfdb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
