[graylog2] Re: graylog2 timestamp not from application log message
Hi Wayne, On Tuesday, 18 October 2016 20:01:11 UTC+2, Wayne wrote: > > The problem is that when an alert email is sent, the Date is showing UTC > time. > Yes, that's intentional. The alert emails aren't linked to any Graylog user, so it's not possible to use the configured timezone of any Graylog user to transform the timestamp of messages in these emails. > Is it something that will be fixed later? > That's rather unlikely. Cheers, Jochen -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/7c315df8-3c0f-42bb-9c45-468d432b5788%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: graylog2 timestamp not from application log message
Hi Wayne, On Tuesday, 18 October 2016 17:04:34 UTC+2, Wayne wrote: > > The converted timestamp: 2016-10-18 15:01:34.559 > and the real timestamp from application log is: 2016-10-18 11:01:34:559 > > There is a four hour difference (when the timezone is configured as either > "Toronto" or "GMT+4" > That sounds correct to me. Mind that the timestamp is always being stored in UTC and only adapted to the user's configured timezone on display. See https://github.com/Graylog2/graylog2-server/issues/2689 for a related issue on GitHub. Cheers, Jochen -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/870eea2a-fa97-4b76-9c12-e057238bbcd8%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: graylog2 timestamp not from application log message
Hi Jochen, I tried again. It looks like the timezone field needs to be filled in. if left blank, no messages will be shipped to graylog server. However, I tried "Toronto", "GMT+4". Both did not fix the timezone issue with timestamp having correct minutes/seconds/milliseconds, but not hours. When I use "GMT-4", messages did not get shipped in. For example, the best case is like: The converted timestamp: 2016-10-18 15:01:34.559 and the real timestamp from application log is: 2016-10-18 11:01:34:559 There is a four hour difference (when the timezone is configured as either "Toronto" or "GMT+4" What is correct timezone setting that can fix this issue? Thanks Wayne On Tuesday, October 18, 2016 at 10:35:58 AM UTC-4, Wayne wrote: > > Hi Jochen, > > It is tricky. > > Now I found out the extractor to overwrite the timestamp actually stopped > the messages to come to graylog server. Once I delete it or rename the > "store as field" to names other than timestamp, the messages come into > graylog server again, but then I could not overwrite the timestamp field. > > I remember there was a brief time I was able to overwrite the field, but > with a different timezone. However, I could not overwrite it at all now. > > What could be some common reasons that prevent messages to come into > graylog if the timestamp field is overwritten? > > Thanks, > > Wayne > > > On Monday, October 17, 2016 at 2:25:53 AM UTC-4, Jochen Schalanda wrote: >> >> Hi Wayne >> >> On Friday, 14 October 2016 19:36:17 UTC+2, Wayne wrote: >>> >>> I have tried your extractor, and it looks like it almost worked, except >>> that the timestamp seems to use UTC, instead of my local time zone. >>> >> >> The date converter can be configured to use a specific timezone. >> >> Cheers, >> Jochen >> > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/61f9136a-6397-4a0b-9676-b4ed7b8436f6%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: graylog2 timestamp not from application log message
Hi Jochen, It is tricky. Now I found out the extractor to overwrite the timestamp actually stopped the messages to come to graylog server. Once I delete it or rename the "store as field" to names other than timestamp, the messages come into graylog server again, but then I could not overwrite the timestamp field. I remember there was a brief time I was able to overwrite the field, but with a different timezone. However, I could not overwrite it at all now. What could be some common reasons that prevent messages to come into graylog if the timestamp field is overwritten? Thanks, Wayne On Monday, October 17, 2016 at 2:25:53 AM UTC-4, Jochen Schalanda wrote: > > Hi Wayne > > On Friday, 14 October 2016 19:36:17 UTC+2, Wayne wrote: >> >> I have tried your extractor, and it looks like it almost worked, except >> that the timestamp seems to use UTC, instead of my local time zone. >> > > The date converter can be configured to use a specific timezone. > > Cheers, > Jochen > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/b75427ee-f04f-4016-b44a-721952b270c3%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: graylog2 timestamp not from application log message
Hi Wayne On Friday, 14 October 2016 19:36:17 UTC+2, Wayne wrote: > > I have tried your extractor, and it looks like it almost worked, except > that the timestamp seems to use UTC, instead of my local time zone. > The date converter can be configured to use a specific timezone. Cheers, Jochen -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/74b34aef-e64a-46be-9523-a3324a171e02%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: graylog2 timestamp not from application log message
Hi Jochen,
I have tried your extractor, and it looks like it almost worked, except
that the timestamp seems to use UTC, instead of my local time zone.
So the timestamp in my case (Toronto) is 4 hours ahead of the timestamp in
the application log.
What is the timezone that I should use? It seems that the Toronto in the
dropdown did not work.
Thanks,
Wayne
On Friday, October 14, 2016 at 12:32:44 PM UTC-4, Jochen Schalanda wrote:
>
> Hi Wayne,
>
> the following extractor is working for me without problem:
>
> {
> "extractors": [
> {
> "title": "Timestamp",
> "extractor_type": "regex",
> "converters": [
> {
> "type": "date",
> "config": {
> "date_format": "-MM-dd HH:mm:ss,SSS",
> "time_zone": "Etc/GMT+2"
> }
> }
> ],
> "order": 0,
> "cursor_strategy": "copy",
> "source_field": "message",
> "target_field": "timestamp",
> "extractor_config": {
> "regex_value": "^([0-9]{4}-[0-9]{2}-[0-9]{2}
> [0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3})"
> },
> "condition_type": "none",
> "condition_value": ""
> }
> ],
> "version": "2.1.1"
> }
>
>
> Cheers,
> Jochen
>
> On Thursday, 13 October 2016 18:41:13 UTC+2, Wayne wrote:
>>
>> Hi Jochen,
>>
>> Just to add a bit more detail:
>>
>> The timestamp in my server log is of the following pattern:
>>
>> 2016-10-13 12:37:00,022
>>
>> I was not able to configure an extractor to extract it as a date type
>> with the pattern like
>> -MM-dd HH:mm:ss,SSS
>>
>> Note: I was creating an Extractor with type of Grok pattern
>>
>>
>> Thanks,
>>
>> Wayne
>>
>>
>> On Thursday, October 13, 2016 at 10:34:29 AM UTC-4, Jochen Schalanda
>> wrote:
>>>
>>> Hi Wayne,
>>>
>>> On Thursday, 13 October 2016 16:30:18 UTC+2, Wayne wrote:
I understand that the timestamp reflects the time that graylog imported
the log messages, and not the timestamp associated with the application
log
message. For example, if I send a log file from my application server to
graylog server, the timestamp of my application log message is a different
field (when extracted) in graylog UI
>>>
>>> Graylog is only falling-back to the ingestion time if the message itself
>>> doesn't include a timestamp or includes an invalid timestamp.
>>>
>>> For example if you're using a GELF input and the GELF messages contain a
>>> valid timestamp field, that timestamp is being used as message
>>> timestamp in Graylog.
>>>
>>>
>>> Is there a workaround?
>>>
>>> What exactly is the problem you're trying to solve?
>>>
>>> Cheers,
>>> Jochen
>>>
>>
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/a5a09f04-feff-4657-8cbb-f266abf24a77%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: graylog2 timestamp not from application log message
Hi Wayne,
the following extractor is working for me without problem:
{
"extractors": [
{
"title": "Timestamp",
"extractor_type": "regex",
"converters": [
{
"type": "date",
"config": {
"date_format": "-MM-dd HH:mm:ss,SSS",
"time_zone": "Etc/GMT+2"
}
}
],
"order": 0,
"cursor_strategy": "copy",
"source_field": "message",
"target_field": "timestamp",
"extractor_config": {
"regex_value": "^([0-9]{4}-[0-9]{2}-[0-9]{2}
[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3})"
},
"condition_type": "none",
"condition_value": ""
}
],
"version": "2.1.1"
}
Cheers,
Jochen
On Thursday, 13 October 2016 18:41:13 UTC+2, Wayne wrote:
>
> Hi Jochen,
>
> Just to add a bit more detail:
>
> The timestamp in my server log is of the following pattern:
>
> 2016-10-13 12:37:00,022
>
> I was not able to configure an extractor to extract it as a date type with
> the pattern like
> -MM-dd HH:mm:ss,SSS
>
> Note: I was creating an Extractor with type of Grok pattern
>
>
> Thanks,
>
> Wayne
>
>
> On Thursday, October 13, 2016 at 10:34:29 AM UTC-4, Jochen Schalanda wrote:
>>
>> Hi Wayne,
>>
>> On Thursday, 13 October 2016 16:30:18 UTC+2, Wayne wrote:
>>>
>>> I understand that the timestamp reflects the time that graylog imported
>>> the log messages, and not the timestamp associated with the application log
>>> message. For example, if I send a log file from my application server to
>>> graylog server, the timestamp of my application log message is a different
>>> field (when extracted) in graylog UI
>>>
>>
>> Graylog is only falling-back to the ingestion time if the message itself
>> doesn't include a timestamp or includes an invalid timestamp.
>>
>> For example if you're using a GELF input and the GELF messages contain a
>> valid timestamp field, that timestamp is being used as message timestamp
>> in Graylog.
>>
>>
>> Is there a workaround?
>>>
>>
>> What exactly is the problem you're trying to solve?
>>
>> Cheers,
>> Jochen
>>
>
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/f66f3f79-265e-40d9-b8f1-a283ba1f2b96%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: graylog2 timestamp not from application log message
Hi Jochen, Just to add a bit more detail: The timestamp in my server log is of the following pattern: 2016-10-13 12:37:00,022 I was not able to configure an extractor to extract it as a date type with the pattern like -MM-dd HH:mm:ss,SSS Note: I was creating an Extractor with type of Grok pattern Thanks, Wayne On Thursday, October 13, 2016 at 10:34:29 AM UTC-4, Jochen Schalanda wrote: > > Hi Wayne, > > On Thursday, 13 October 2016 16:30:18 UTC+2, Wayne wrote: >> >> I understand that the timestamp reflects the time that graylog imported >> the log messages, and not the timestamp associated with the application log >> message. For example, if I send a log file from my application server to >> graylog server, the timestamp of my application log message is a different >> field (when extracted) in graylog UI >> > > Graylog is only falling-back to the ingestion time if the message itself > doesn't include a timestamp or includes an invalid timestamp. > > For example if you're using a GELF input and the GELF messages contain a > valid timestamp field, that timestamp is being used as message timestamp > in Graylog. > > > Is there a workaround? >> > > What exactly is the problem you're trying to solve? > > Cheers, > Jochen > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/1d5ceb86-2b2c-4509-9287-51bcb87cccbe%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: graylog2 timestamp not from application log message
Hi Jochen, I installed the "Graylog collector sidecar" in a server node to send the tail of the log file to Graylog2 server in another machine. In the UI of Graylog2 server, I created an Extractor (Grok pattern) to generate new fields such as log level, log message, and mytimestamp. The mytimestamp is by default a string type, so I create another Extractor (copy input) to create another field mytimestampDate. I also load the custom mapping so that mytimestampDate will be date type. I tried to modify the field name mytimestampDate to timestamp, However, messages did not get through Graylog2 server, and the timestamp in Graylog2 is still UTC time. Is it not the right way to get the log messages into Graylog2 server? Thanks, Wayne On Thursday, October 13, 2016 at 10:34:29 AM UTC-4, Jochen Schalanda wrote: > > Hi Wayne, > > On Thursday, 13 October 2016 16:30:18 UTC+2, Wayne wrote: >> >> I understand that the timestamp reflects the time that graylog imported >> the log messages, and not the timestamp associated with the application log >> message. For example, if I send a log file from my application server to >> graylog server, the timestamp of my application log message is a different >> field (when extracted) in graylog UI >> > > Graylog is only falling-back to the ingestion time if the message itself > doesn't include a timestamp or includes an invalid timestamp. > > For example if you're using a GELF input and the GELF messages contain a > valid timestamp field, that timestamp is being used as message timestamp > in Graylog. > > > Is there a workaround? >> > > What exactly is the problem you're trying to solve? > > Cheers, > Jochen > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/34c8485f-08d8-4d94-905a-d126ced35e09%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: graylog2 timestamp not from application log message
Hi Wayne, On Thursday, 13 October 2016 16:30:18 UTC+2, Wayne wrote: > > I understand that the timestamp reflects the time that graylog imported > the log messages, and not the timestamp associated with the application log > message. For example, if I send a log file from my application server to > graylog server, the timestamp of my application log message is a different > field (when extracted) in graylog UI > Graylog is only falling-back to the ingestion time if the message itself doesn't include a timestamp or includes an invalid timestamp. For example if you're using a GELF input and the GELF messages contain a valid timestamp field, that timestamp is being used as message timestamp in Graylog. Is there a workaround? > What exactly is the problem you're trying to solve? Cheers, Jochen -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/f3b1331d-8b10-4e7a-97c4-157e7193ab53%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
