[graylog2] Re: TimeStamp

[Jochen Schalanda]

Hi,

you have to use some filter (e. g. a Grok extractor with the pattern 
%{COMMONAPACHELOG}) to extract the date from your web server access logs 
and substitute the timestamp field of the message.

Cheers,
Jochen

On Wednesday, 23 November 2016 07:15:30 UTC+1, suj...@bw.ae wrote:
>
> Hello,
>
> I have few servers. Graylog is fetching logs from those server. Graylog is 
> showing correct timestamp of the logs which is generated those server. But 
> there is difference of 9 OR 6 OR 4 seconds.
>
> In below example you can see the first timestamp is showing 6 seconds more 
> than the log
>
> 2016-11-23 09:53:09.000 server 
> server1 AccesLogs xx.xx.xx.xx. 5 - - [23/Nov/*2016:09:53:03* +0400] "GET 
> /sdfdsfdsf/sdfsdfdsf/sdfdsfsdfdsfds.js HTTP/1.1" 200 4343
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/5fac8e67-b9ca-4464-a57d-fbb847981e18%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Timestamp and event time mismatch

[kathleen . boht]

Thank you, Jochen.  I'll dig into this, hopefully I'll get it figured out! 
 New to graylog & extractors.

Kathleen

On Friday, September 9, 2016 at 9:55:49 AM UTC-5, Jochen Schalanda wrote:
>
> Hi,
>
> you can extract the timestamp from your "Event Data" and override the 
> message timestamp using extractors: 
> http://docs.graylog.org/en/2.1/pages/extractors.html
>
> Cheers,
> Jochen
>
> On Friday, 9 September 2016 16:21:12 UTC+2, kathle...@gmail.com wrote:
>>
>> Hello There,
>>
>>
>> Concerning log ingestion time stamps, we notice that the log entry 
>> timestamp and the event time stamp stamp don't quite match. At this point 
>> it really only seems to be off by a few milliseconds. I'm assuming that the 
>> log time stamp is the time the log was ingested into graylog (we used nxlog 
>> with collector-sidecar). Just wanted to check in to see if there is 
>> something in our configuration we could have done differently to tighten up 
>> the difference?
>>
>>
>>
>> 
>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/ce502eba-c1d3-475d-8426-04cd0dc98f51%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Timestamp field depecrated.

[Jochen Schalanda]

Hi Juan,

Graylog 1.x doesn't support Elasticsearch 2.0.x, see 
https://github.com/Graylog2/graylog2-server/issues/1518.

Cheers,
Jochen

On Monday, 16 November 2015 19:29:18 UTC+1, Juan Andres Ramirez wrote:
>
> Hello Guys,
> I tried migrate my indexes from elasticsearch 1.7 to 2.0, but I 
> have a lot problem with _timestamp field, and I can't migrate the indexes 
> because I have this error:
>
>  "error" : {
> "root_cause" : [ {
>   "type" : "snapshot_restore_exception",
>   "reason" : "[my_backup:snapshot_1] cannot restore index 
> [graylog2_19] because it cannot be upgraded"
> } ],
> "type" : "snapshot_restore_exception",
> "reason" : "[my_backup:snapshot_1] cannot restore index [graylog2_19] 
> because it cannot be upgraded",
> "caused_by" : {
>   "type" : "illegal_state_exception",
>   "reason" : "unable to upgrade the mappings for the index 
> [graylog2_19], reason: [Mapper for [_timestamp] conflicts with existing 
> mapping in other types:\n[mapper [_timestamp] is used by multiple types. 
> Set update_all_types to true to update [format] across all types.]]",
>   "caused_by" : {
> "type" : "illegal_argument_exception",
> "reason" : "Mapper for [_timestamp] conflicts with existing 
> mapping in other types:\n[mapper [_timestamp] is used by multiple types. 
> Set update_all_types to true to update [format] across all types.]"
>   }
> }
>   },
>   "status" : 500
>
>
> This is index number 19 and I have 50...I can't lose my data, anyone knows 
> how to fix it?.
>
> Link elasticsearch about this topic: 
> https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-timestamp-field.html
>
>
> Thank you.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/6c2896fe-765d-4b31-a560-cf553cbe1f90%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Timestamp fixup

[Sandro Roth]

Hi Jochen

There is only one index, but to be sure I ran the script with -i '*'. Same 
result, didn't change any documents.

Let me know if you need some logs or debug outputs.

Thanks
Sandro

-- 
You received this message because you are subscribed to the Google Groups 
graylog2 group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Timestamp fixup

[Jochen Schalanda]

Hello Sandro,

it looks like you've only run the tool for a single Elasticsearch index 
(graylog2_0). Graylog2 usually is using multiple indices for storing 
messages (which can be configured in the retention settings in your 
graylog2.conf). It's very likely that the messages with the defective 
timestamps are stored in one of the other indices.

I would suggest that you run the fixup tool for all the remaining indices 
as well. If the graphs still start at 1/1/1970 after that, we'll have to 
investigate further.


Cheers,
Jochen 

Am Freitag, 21. November 2014 11:28:29 UTC+1 schrieb Sandro Roth:

 Hi there

 We upgraded graylog and elasticsearch from 0.20.2 and 0.90.10 to 0.91.3 
 and 1.3.4.
 After the upgrade we noticed that searching through everything (all 
 messages) results in a graph that starts on Jan 1st 1970. (see attachment)
 I remembered reading about this in the release notes so I went ahead and 
 ran the fixup script on our test setup.

  # ./graylog2-es-timestamp-fixup -F  -i graylog2_0
 2014-11-21 11:06:16,428 WARN : org.elasticsearch.discovery - [spch9320] 
 waited for 3s and no initial state was set by the discovery
 2014-11-21 11:06:17,135 INFO : org.graylog2.ESTimestampFixup - Changed 0 
 of total 22579 documents (8.86% checked)
 2014-11-21 11:06:17,440 INFO : org.graylog2.ESTimestampFixup - Changed 0 
 of total 22579 documents (17.72% checked)
 2014-11-21 11:06:17,676 INFO : org.graylog2.ESTimestampFixup - Changed 0 
 of total 22579 documents (26.57% checked)
 2014-11-21 11:06:17,866 INFO : org.graylog2.ESTimestampFixup - Changed 0 
 of total 22579 documents (35.43% checked)
 2014-11-21 11:06:18,021 INFO : org.graylog2.ESTimestampFixup - Changed 0 
 of total 22579 documents (44.29% checked)
 2014-11-21 11:06:18,141 INFO : org.graylog2.ESTimestampFixup - Changed 0 
 of total 22579 documents (53.15% checked)
 2014-11-21 11:06:18,244 INFO : org.graylog2.ESTimestampFixup - Changed 0 
 of total 22579 documents (62.00% checked)
 2014-11-21 11:06:18,356 INFO : org.graylog2.ESTimestampFixup - Changed 0 
 of total 22579 documents (70.86% checked)
 2014-11-21 11:06:18,431 INFO : org.graylog2.ESTimestampFixup - Changed 0 
 of total 22579 documents (79.72% checked)
 2014-11-21 11:06:18,510 INFO : org.graylog2.ESTimestampFixup - Changed 0 
 of total 22579 documents (88.58% checked)
 2014-11-21 11:06:18,586 INFO : org.graylog2.ESTimestampFixup - Changed 0 
 of total 22579 documents (97.44% checked)
 2014-11-21 11:06:18,616 INFO : org.graylog2.ESTimestampFixup - Changed 0 
 of total 22579 documents (100.00% checked)


 So it didn't change anything in the index, why not? The problem is still 
 there in graylog..
 Am I missing something?


 Thanks for your help
 Regards


-- 
You received this message because you are subscribed to the Google Groups 
graylog2 group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.