[graylog2] Re: Graylog server always collect expired logs, these logs are generated long before , and now the switch has no such logs.

[ql . wayne]

Hi,
 
I haved stopped input, the graylog should not receive all logs, BUT the 
abnormal message can be received as before. 

在 2017年2月6日星期一 UTC+8下午6:40:50，Jochen Schalanda写道：
>
> Hi,
>
> are you sure that these messages are ingested right now and don't simply 
> have a timestamp "in the future" (e. g. because of timezone issues) and 
> have been ingested some hours ago?
>
> Cheers,
> Jochen
>
> On Monday, 6 February 2017 11:17:19 UTC+1, ql.w...@163.com wrote:
>>
>> Hi,
>> This messages shows received by deleted input on  0de4fb00 / Unknown, as 
>> shown in FIG:
>>
>>
>> 
>>
>> But the normal messages shows received by netsyslog on  0de4fb00 / 
>> Unknown,as shown in FIG:
>>
>>
>> 
>>
>>
>> 在 2017年2月6日星期一 UTC+8下午5:11:55，Jochen Schalanda写道：
>>>
>>> Hi,
>>>
>>> when you click on one of these messages, you can see on which input they 
>>> were received next to the "Received by" field.
>>>
>>> Once you have identified the input, you can use tools like Wireshark, 
>>> tcpdump, or simply lsof to identify where these messages come from.
>>>
>>> Cheers,
>>> Jochen
>>>
>>>
>>> On Monday, 6 February 2017 04:06:00 UTC+1, ql.w...@163.com wrote:

 Hi,

 I deleted the command that send logs to graylog server in the switch, 
 But, graylog can receive the logs of this switch as before. I don't know 
 where those logs received by the graylog server come from?  


 


 The switch do not send logs to graylog,  But, graylog can receive the 
 logs of this switch as before. As shown in FIG. 



 在 2017年2月4日星期六 UTC+8下午6:07:06，Jochen Schalanda写道：
>
> Hi,
>
> please elaborate on your problem. I'm not sure what you're trying to 
> say.
>
> What did you expect to happen or retrieve? What did actually happen?
> As far as I see, the timestamps of the log messages are correct.
>
> Cheers,
> Jochen
>
> On Saturday, 4 February 2017 10:48:25 UTC+1, ql.w...@163.com wrote:
>>
>> My graylog server always collect expired logs, these logs are 
>> generated long before , and now the switch has no such logs.
>> [image: image] 
>> 
>>
>> The current log's source is 2017, The log whose source is 
>> G1-K115-ACC-SW-48 is very early, but the server is collecting now.
>>
>> This problem has troubled me for weeks. How to solve this problem?
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/e840d133-4d6c-4dfd-adbc-aa90eb2dd6ba%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Graylog server always collect expired logs, these logs are generated long before , and now the switch has no such logs.

[ql . wayne]

Hi,
This messages shows received by deleted input on  0de4fb00 / Unknown, as 
shown in FIG:



But the normal messages shows received by netsyslog on  0de4fb00 / 
Unknown,as shown in FIG:




在 2017年2月6日星期一 UTC+8下午5:11:55，Jochen Schalanda写道：
>
> Hi,
>
> when you click on one of these messages, you can see on which input they 
> were received next to the "Received by" field.
>
> Once you have identified the input, you can use tools like Wireshark, 
> tcpdump, or simply lsof to identify where these messages come from.
>
> Cheers,
> Jochen
>
>
> On Monday, 6 February 2017 04:06:00 UTC+1, ql.w...@163.com wrote:
>>
>> Hi,
>>
>> I deleted the command that send logs to graylog server in the switch, 
>> But, graylog can receive the logs of this switch as before. I don't know 
>> where those logs received by the graylog server come from?  
>>
>>
>> 
>>
>>
>> The switch do not send logs to graylog,  But, graylog can receive the 
>> logs of this switch as before. As shown in FIG. 
>>
>>
>>
>> 在 2017年2月4日星期六 UTC+8下午6:07:06，Jochen Schalanda写道：
>>>
>>> Hi,
>>>
>>> please elaborate on your problem. I'm not sure what you're trying to say.
>>>
>>> What did you expect to happen or retrieve? What did actually happen?
>>> As far as I see, the timestamps of the log messages are correct.
>>>
>>> Cheers,
>>> Jochen
>>>
>>> On Saturday, 4 February 2017 10:48:25 UTC+1, ql.w...@163.com wrote:

 My graylog server always collect expired logs, these logs are generated 
 long before , and now the switch has no such logs.
 [image: image] 
 

 The current log's source is 2017, The log whose source is 
 G1-K115-ACC-SW-48 is very early, but the server is collecting now.

 This problem has troubled me for weeks. How to solve this problem?

>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/4f350e28-c425-48e0-ab78-5d14ed81ddaa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Graylog server always collect expired logs, these logs are generated long before , and now the switch has no such logs.

[ql . wayne]

Hi,

I deleted the command that send logs to graylog server in the switch, But, 
graylog can receive the logs of this switch as before. I don't know where 
those logs received by the graylog server come from?  




The switch do not send logs to graylog,  But, graylog can receive the logs 
of this switch as before. As shown in FIG. 



在 2017年2月4日星期六 UTC+8下午6:07:06，Jochen Schalanda写道：
>
> Hi,
>
> please elaborate on your problem. I'm not sure what you're trying to say.
>
> What did you expect to happen or retrieve? What did actually happen?
> As far as I see, the timestamps of the log messages are correct.
>
> Cheers,
> Jochen
>
> On Saturday, 4 February 2017 10:48:25 UTC+1, ql.w...@163.com wrote:
>>
>> My graylog server always collect expired logs, these logs are generated 
>> long before , and now the switch has no such logs.
>> [image: image] 
>> 
>>
>> The current log's source is 2017, The log whose source is 
>> G1-K115-ACC-SW-48 is very early, but the server is collecting now.
>>
>> This problem has troubled me for weeks. How to solve this problem?
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/4d41af0f-a8ab-45bb-89c9-c03602ee4788%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Graylog server always collect expired logs, these logs are generated long before , and now the switch has no such logs.

[ql . wayne]

My graylog server always collect expired logs, these logs are generated 
long before , and now the switch has no such logs.
[image: image] 


The current log's source is 2017, The log whose source is G1-K115-ACC-SW-48 
is very early, but the server is collecting now.

This problem has troubled me for weeks. How to solve this problem?

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/a5704476-8501-4371-81e6-dc1339e8d9f8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: re-index after the data mapping is changed with updated Extractor.

[Wayne]

Thank you very much Jochen.

I will investigate the solution later.

Thanks,

Wayne

On Friday, October 21, 2016 at 11:54:43 AM UTC-4, Jochen Schalanda wrote:
>
> Hi Wayne,
>
> On Friday, 21 October 2016 15:44:06 UTC+2, Wayne wrote:
>>
>> In the case where I install Elastic search and Graylog in the same 
>> machine, the graylog-x is basically the Elasticsearch index, if I install 
>> Logstash and configure the Elastic search as input and Graylog as log 
>> server to receive input, isn't this an endless loop because the input and 
>> destination are the same search index?
>>
>
> Ideally, you would read from an older index (e. g. "graylog_0") into the 
> current write-active index (e. g. "graylog_1"). 
>
> Cheers,
> Jochen
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/87c9ba23-6b7b-4932-8470-a3589cc18efe%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: re-index after the data mapping is changed with updated Extractor.

[Wayne]

Hi Jochen,

I just realized that I was investigating Fluentd before, and I forgot to 
disable the td-agent that sends data to ElasticSearch. I guess the 
additional indexes were generated because this reason. I have now disabled 
it and hope to see that the additional indexes will not be generated.

As for the re-indexing option, I would like to get more information on how 
to do it properly since there is no out-of-box solution from Graylog, and 
it is a common concern due to addition or change of data/index mapping. So 
in order to do re-index, I need to install logstash and configure the input 
as ElasticSearch instance and output to Graylog instance. 

In the case where I install Elastic search and Graylog in the same machine, 
the graylog-x is basically the Elasticsearch index, if I install Logstash 
and configure the Elastic search as input and Graylog as log server to 
receive input, isn't this an endless loop because the input and destination 
are the same search index?

Thanks,

Wayne



On Friday, October 21, 2016 at 9:02:12 AM UTC-4, Jochen Schalanda wrote:
>
> Hi Wayne,
>
> On Friday, 21 October 2016 14:51:55 UTC+2, Wayne wrote:
>>
>> I only installed Graylog2 server, Elasticsearch, Mongodb based on the 
>> lastest Graylog2 document. The daily logstash-.MM.dd was generated, but 
>> I did not install Logstash. Is this normal?
>>
>
> No, at least that index definitely hasn't been created or touched in any 
> way by Graylog.
>
>
> As far as the option of re-indexing you mentionded, are you saying I can 
>> use the ElasticSearch instance as input, and use log shipper such as 
>> graylog collector sidecar to push the index to graylog server? My concern 
>> is that would duplicate the data. In addition, can graylog collector 
>> sidecar be log shipper in this scenario, or I need to install logstash to 
>> do the job?
>>
>
> Yes, it would naturally duplicate the data and yes, you need Logstash (or 
> any other program being able to read from Elasticsearch and send output to 
> Graylog via GELF) for that. It's not possible to do this with the Graylog 
> Collector Sidecar.
>
>
> Cheers,
> Jochen 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/a9206a2e-d637-4433-8d49-0a74da51f94e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: re-index after the data mapping is changed with updated Extractor.

[Wayne]

Hi Jochen,

To clarify the question:

(1) About the logstash-.MM.dd indexes

I only installed Graylog2 server, Elasticsearch, Mongodb based on the 
lastest Graylog2 document. The daily logstash-.MM.dd was generated, but 
I did not install Logstash. Is this normal?

(2) About the re-indexing option
As far as the option of re-indexing you mentionded, are you saying I can 
use the ElasticSearch instance as input, and use log shipper such as 
graylog collector sidecar to push the index to graylog server? My concern 
is that would duplicate the data. In addition, can graylog collector 
sidecar be log shipper in this scenario, or I need to install logstash to 
do the job?

Thanks

Wayne



On Thursday, October 20, 2016 at 1:54:53 PM UTC-4, Jochen Schalanda wrote:
>
> Hi Wayne,
>
> On Thursday, 20 October 2016 18:13:21 UTC+2, Wayne wrote:
>>
>> That probably requires setup of additional Graylog server plus installing 
>> logstach as log shipper?
>>
>
> No, you can read from the same Elasticsearch cluster and write into the 
> same Graylog instance.
>  
>
> I can see two types of indexes in /var/lib/elasticsearch/graylog/nodes
>>
>> (1) graylog_x
>>
>> (2) logstash-.MM.dd
>>
>> What is the relationship between between these two types of indexes, and 
>> if the configuration is set up to delete old indexes, which indexes will be 
>> deleted?
>>
>
> The first one, graylog_*, is managed by Graylog, the latter is being 
> created and written into by logstash (depending on the configuration).
>
> Graylog doesn't have to do anything with the latter one and can't read 
> from it.
>
>
> Cheers,
> Jochen 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/fc8ad160-b699-4458-853f-59f4e5c5b0ef%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: re-index after the data mapping is changed with updated Extractor.

[Wayne]

Hi Jochen,

That probably requires setup of additional Graylog server plus installing 
logstach as log shipper? It seems to be a bit messy.

Another question:

I can see two types of indexes in /var/lib/elasticsearch/graylog/nodes

(1) graylog_x

(2) logstash-.MM.dd

What is the relationship between between these two types of indexes, and if 
the configuration is set up to delete old indexes, which indexes will be 
deleted?


Thanks,

Wayne

On Thursday, October 20, 2016 at 11:50:08 AM UTC-4, Jochen Schalanda wrote:
>
> Hi Wayne,
>
> On Thursday, 20 October 2016 16:49:23 UTC+2, Wayne wrote:
>>
>> I am interested to know if there is a way to re-index all the data once a 
>> mapping is updated?
>>
>
> Graylog doesn't support this out-of-the-box.
>
> If the solution is not available now, is it in the next release?
>>
>
> No. 
>
> On the other hand, is there anyway to do it manually? I understand that 
>> the ELK stack could do a re-index, but I am not sure if there is a way to 
>> do it similarly?
>>
>
> You can re-index messages using logstash (input from Elasticsearch, output 
> to Graylog).
>  
>
> Cheers,
> Jochen
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/27868742-e4f6-47d6-aaa5-72e2b1c0ea79%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] re-index after the data mapping is changed with updated Extractor.

[Wayne]

Hi All,

I am interested to know if there is a way to re-index all the data once a 
mapping is updated?


I googled it and found out there was no out-of-box solution up to last 
year. I am wondering if Graylog2 has a solution now.


If the solution is not available now, is it in the next release?


On the other hand, is there anyway to do it manually? I understand that the 
ELK stack could do a re-index, but I am not sure if there is a way to do it 
similarly?


Thanks,

Wayne

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/2388f6be-dc44-4ee6-9780-23eec26775ab%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Does Graylog server save a copy of the original log messages before indexing the message

[Wayne]

Hi Jochen,

Just want to explore a bit further.

These messages are now in binary format, and it seems to be parsed already. 
Is there a way to convert them back to original text messages? or there is 
no way to convert it back to original text form?

I am asking the question on behalf of one of my colleague who was thinking 
about retrieving information from the consolidated data (log messages from 
multiple source).

Thanks,

Wayne

On Thursday, October 20, 2016 at 6:16:14 AM UTC-4, Jochen Schalanda wrote:
>
> Hi Wayne,
>
> On Wednesday, 19 October 2016 21:28:25 UTC+2, Wayne wrote:
>>
>> Let's say we send a query and search a couple of records, now we would 
>> like to retrieve the original text message. Does Graylog keep the original 
>> copy of the log message?
>>
>
> No, it doesn't.
>  
>
> In addition, the disk based journal seems to keep some data, but not 
>> completely visible. Are those the copy of the messages?
>>
>
> Basically yes. The disk journal contains the raw binary message received 
> by an input until a codec decodes the message and indexes it into 
> Elasticsearch.
>
> Cheers,
> Jochen 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/e5e308d4-7fab-4952-9d19-859db6f4f1a7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Internal message queue for graylog2?

[Wayne]

I think it is sufficient for us to stick to the default configuration 
without external message queue.

Thanks,

Wayne

On Wednesday, October 19, 2016 at 9:14:11 AM UTC-4, Jochen Schalanda wrote:
>
> Hi Wayne,
>
> On Wednesday, 19 October 2016 15:07:07 UTC+2, Wayne wrote:
>>
>> It is stated in 2.1 document that Kafka and RabbitMQ can be configured as 
>> transport queue.
>>
>> What are the use cases/scenarios which we need to do the above 
>> configuration considering Graylog already has its own way to persist the 
>> messages?
>>
>
> It can be useful for connecting offsite locations with bad network 
> connection or if log messages aren't exclusively consumed by Graylog.
>
> If you can't come up with a use case for using a message broker like 
> RabbitMQ or Apache Kafka, it's probably not necessary for you…
>
> Cheers,
> Jochen
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/e8bdc639-f8ba-4d1d-a034-78c7bdd150c8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Does Graylog server save a copy of the original log messages before indexing the message

[Wayne]

Hi All,

Let's say we send a query and search a couple of records, now we would like 
to retrieve the original text message. Does Graylog keep the original copy 
of the log message?

In addition, the disk based journal seems to keep some data, but not 
completely visible. Are those the copy of the messages?

Thanks,

Wayne



-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/f43cf707-2941-4f10-bf31-ca077f7da68b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Some fields generated from Extractor are not searchable

[Wayne]

Hi Jochen,

What is strange about it is that the "Stream" rules apparently work with 
the field "log_message", but a search query does not work. 

I send a raw Elastic query and still not much information about why it is 
not working.

The custom mapping is useful if the data type is not the default string 
type. However, the log_message field is still string type. So it may not 
make much difference if I set up custom mapping for this field?

Thanks,

Wayne


On Wednesday, October 19, 2016 at 12:22:32 PM UTC-4, Jochen Schalanda wrote:
>
> Hi Wayne,
>
> On Wednesday, 19 October 2016 17:36:10 UTC+2, Wayne wrote:
>>
>> Is there additional configuration that is required to ensure all the 
>> extracted fields to be searchable?
>>
>
> See 
> http://docs.graylog.org/en/2.1/pages/configuration/elasticsearch.html#custom-index-mappings
>  
> for details.
>
> Cheers,
> Jochen 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/8876b4c5-a05a-4f58-8810-164570aec67d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Some fields generated from Extractor are not searchable

[Wayne]

Hi All,

I configured a couple of Extractors to extract fields from the log message. 
Some fields can be searched, but others can not be searched.


Example:

I have a field called "level" (log level) and it can be searched. I can 
also see this field listed as a property in search index 
logstash-.MM.dd. I have another field called "log_message" and it is 
not searchable. When I checked the mapping, it is not listed as a property 
in logstash-.MM.dd. 

When I check the mapping in search index graylog-x, both are listed.

The failed search example:

If I use message field, I can search a record with a string Exception in 
the message within 2 hours of time frame, but if I use log_message field 
(remove the timestamp part and contains the string Exception), I can not 
search the record although the string is in the log_message field.

Is there additional configuration that is required to ensure all the 
extracted fields to be searchable?

Thanks,

Wayne


Note:

I access url to check the fields and mapping in each search index:

http://localhost:9200/_mappings

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/7ffe7dcd-9a0d-4ee5-a099-9d7d40f20f7b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Internal message queue for graylog2?

[Wayne]

Hi Jochen,

It is stated in 2.1 document that Kafka and RabbitMQ can be configured as 
transport queue.

What are the use cases/scenarios which we need to do the above 
configuration considering Graylog already has its own way to persist the 
messages? 

Thanks,

Wayne


On Wednesday, October 19, 2016 at 6:48:05 AM UTC-4, Jochen Schalanda wrote:
>
> Hi Wayne,
>
> Graylog writes messages into a disk journal once they have been received 
> and will only remove them from the journal again, if they've been 
> successfully been indexed into Elasticsearch.
>
> Cheers,
> Jochen
>
> On Tuesday, 18 October 2016 18:41:50 UTC+2, Wayne wrote:
>>
>> Hi All,
>>
>> I would like to understand how Graylog is trying to send message without 
>> additional configuration with kafka or RabbitMQ. 
>>
>> I am currently using Graylog collector sidebar to configure filebeat to 
>> send the tail of application log messages to Graylog server, and I am not 
>> sure if there is any internal message queue to hold messages in case of 
>> high load.
>>
>>
>> Thanks,
>>
>> Wayne
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/4c84b271-98da-4e86-ba16-63c16f196a26%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Internal message queue for graylog2?

[Wayne]

Hi All,

I would like to understand how Graylog is trying to send message without 
additional configuration with kafka or RabbitMQ. 

I am currently using Graylog collector sidebar to configure filebeat to 
send the tail of application log messages to Graylog server, and I am not 
sure if there is any internal message queue to hold messages in case of 
high load.


Thanks,

Wayne

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/2355f437-d6d5-4bad-80eb-7e57c0fd2715%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: graylog2 timestamp not from application log message

[Wayne]

Hi Jochen,

I tried again. It looks like the timezone field needs to be filled in. if 
left blank, no messages will be shipped to graylog server.

However, I tried "Toronto", "GMT+4". Both did not fix the timezone issue 
with timestamp having correct minutes/seconds/milliseconds, but not hours. 
When I use "GMT-4", messages did not get shipped in.

For example, the best case is like:

The converted timestamp: 2016-10-18 15:01:34.559
and the real timestamp from application log is: 2016-10-18 11:01:34:559

There is a four hour difference (when the timezone is configured as either 
"Toronto" or "GMT+4"

What is correct timezone setting that can fix this issue?

Thanks

Wayne


On Tuesday, October 18, 2016 at 10:35:58 AM UTC-4, Wayne wrote:
>
> Hi Jochen,
>
> It is tricky.
>
> Now I found out the extractor to overwrite the timestamp actually stopped 
> the messages to come to graylog server. Once I delete it or rename the 
> "store as field" to names other than timestamp, the messages come into 
> graylog server again, but then I could not overwrite the timestamp field.
>
> I remember there was a brief time I was able to overwrite the field, but 
> with a different timezone. However, I could not overwrite it at all now.
>
> What could be some common reasons that prevent messages to come into 
> graylog if the timestamp field is overwritten?
>
> Thanks,
>
> Wayne
>
>
> On Monday, October 17, 2016 at 2:25:53 AM UTC-4, Jochen Schalanda wrote:
>>
>> Hi Wayne
>>
>> On Friday, 14 October 2016 19:36:17 UTC+2, Wayne wrote:
>>>
>>> I have tried your extractor, and it looks like it almost worked, except 
>>> that the timestamp seems to use UTC, instead of my local time zone.
>>>
>>
>> The date converter can be configured to use a specific timezone.
>>
>> Cheers,
>> Jochen
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/61f9136a-6397-4a0b-9676-b4ed7b8436f6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: graylog2 timestamp not from application log message

[Wayne]

Hi Jochen,

It is tricky.

Now I found out the extractor to overwrite the timestamp actually stopped 
the messages to come to graylog server. Once I delete it or rename the 
"store as field" to names other than timestamp, the messages come into 
graylog server again, but then I could not overwrite the timestamp field.

I remember there was a brief time I was able to overwrite the field, but 
with a different timezone. However, I could not overwrite it at all now.

What could be some common reasons that prevent messages to come into 
graylog if the timestamp field is overwritten?

Thanks,

Wayne


On Monday, October 17, 2016 at 2:25:53 AM UTC-4, Jochen Schalanda wrote:
>
> Hi Wayne
>
> On Friday, 14 October 2016 19:36:17 UTC+2, Wayne wrote:
>>
>> I have tried your extractor, and it looks like it almost worked, except 
>> that the timestamp seems to use UTC, instead of my local time zone.
>>
>
> The date converter can be configured to use a specific timezone.
>
> Cheers,
> Jochen
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/b75427ee-f04f-4016-b44a-721952b270c3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: graylog2 timestamp not from application log message

[Wayne]

Hi Jochen,

I have tried your extractor, and it looks like it almost worked, except 
that the timestamp seems to use UTC, instead of my local time zone.

So the timestamp in my case (Toronto) is 4 hours ahead of the timestamp in 
the application log.

What is the timezone that I should use? It seems that the Toronto in the 
dropdown did not work.

Thanks,

Wayne



On Friday, October 14, 2016 at 12:32:44 PM UTC-4, Jochen Schalanda wrote:
>
> Hi Wayne,
>
> the following extractor is working for me without problem:
>
> {
>   "extractors": [
> {
>   "title": "Timestamp",
>   "extractor_type": "regex",
>   "converters": [
> {
>   "type": "date",
>   "config": {
> "date_format": "-MM-dd HH:mm:ss,SSS",
> "time_zone": "Etc/GMT+2"
>   }
> }
>   ],
>   "order": 0,
>   "cursor_strategy": "copy",
>   "source_field": "message",
>   "target_field": "timestamp",
>   "extractor_config": {
> "regex_value": "^([0-9]{4}-[0-9]{2}-[0-9]{2} 
> [0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3})"
>   },
>   "condition_type": "none",
>   "condition_value": ""
> }
>   ],
>   "version": "2.1.1"
> }
>
>
> Cheers,
> Jochen
>
> On Thursday, 13 October 2016 18:41:13 UTC+2, Wayne wrote:
>>
>> Hi Jochen,
>>
>> Just to add a bit more detail:
>>
>> The timestamp in my server log is of the following pattern:
>>
>> 2016-10-13 12:37:00,022
>>
>> I was not able to configure an extractor to extract it as a date type 
>> with the pattern like
>> -MM-dd HH:mm:ss,SSS
>>
>> Note: I was creating an Extractor with type of Grok pattern
>>
>>
>> Thanks,
>>
>> Wayne
>>
>>
>> On Thursday, October 13, 2016 at 10:34:29 AM UTC-4, Jochen Schalanda 
>> wrote:
>>>
>>> Hi Wayne,
>>>
>>> On Thursday, 13 October 2016 16:30:18 UTC+2, Wayne wrote:
>>>>
>>>> I understand that the timestamp reflects the time that graylog imported 
>>>> the log messages, and not the timestamp associated with the application 
>>>> log 
>>>> message. For example, if I send a log file from my application server to 
>>>> graylog server, the timestamp of my application log message is a different 
>>>> field (when extracted) in graylog UI
>>>>
>>>
>>> Graylog is only falling-back to the ingestion time if the message itself 
>>> doesn't include a timestamp or includes an invalid timestamp.
>>>
>>> For example if you're using a GELF input and the GELF messages contain a 
>>> valid timestamp field, that timestamp is being used as message 
>>> timestamp in Graylog.
>>>
>>>
>>> Is there a workaround?
>>>>
>>>
>>> What exactly is the problem you're trying to solve? 
>>>
>>> Cheers,
>>> Jochen
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/a5a09f04-feff-4657-8cbb-f266abf24a77%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: graylog2 timestamp not from application log message

[Wayne]

Hi Jochen,

Just to add a bit more detail:

The timestamp in my server log is of the following pattern:

2016-10-13 12:37:00,022

I was not able to configure an extractor to extract it as a date type with 
the pattern like
-MM-dd HH:mm:ss,SSS

Note: I was creating an Extractor with type of Grok pattern


Thanks,

Wayne


On Thursday, October 13, 2016 at 10:34:29 AM UTC-4, Jochen Schalanda wrote:
>
> Hi Wayne,
>
> On Thursday, 13 October 2016 16:30:18 UTC+2, Wayne wrote:
>>
>> I understand that the timestamp reflects the time that graylog imported 
>> the log messages, and not the timestamp associated with the application log 
>> message. For example, if I send a log file from my application server to 
>> graylog server, the timestamp of my application log message is a different 
>> field (when extracted) in graylog UI
>>
>
> Graylog is only falling-back to the ingestion time if the message itself 
> doesn't include a timestamp or includes an invalid timestamp.
>
> For example if you're using a GELF input and the GELF messages contain a 
> valid timestamp field, that timestamp is being used as message timestamp 
> in Graylog.
>
>
> Is there a workaround?
>>
>
> What exactly is the problem you're trying to solve? 
>
> Cheers,
> Jochen
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/1d5ceb86-2b2c-4509-9287-51bcb87cccbe%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: graylog2 timestamp not from application log message

[Wayne]

Hi Jochen,

I installed the "Graylog collector sidecar" in a server node to send the 
tail of the log file to Graylog2 server in another machine.

In the UI of Graylog2 server, I created an Extractor (Grok pattern) to 
generate new fields such as log level, log message, and mytimestamp. The 
mytimestamp is by default a string type, so I create another Extractor 
(copy input) to create another field mytimestampDate. I also load the 
custom mapping so that mytimestampDate will be date type.

I tried to modify the field name mytimestampDate to timestamp, However, 
messages did not get through Graylog2 server, and the timestamp in Graylog2 
is still UTC time.

Is it not the right way to get the log messages into Graylog2 server?

Thanks,

Wayne
 

On Thursday, October 13, 2016 at 10:34:29 AM UTC-4, Jochen Schalanda wrote:
>
> Hi Wayne,
>
> On Thursday, 13 October 2016 16:30:18 UTC+2, Wayne wrote:
>>
>> I understand that the timestamp reflects the time that graylog imported 
>> the log messages, and not the timestamp associated with the application log 
>> message. For example, if I send a log file from my application server to 
>> graylog server, the timestamp of my application log message is a different 
>> field (when extracted) in graylog UI
>>
>
> Graylog is only falling-back to the ingestion time if the message itself 
> doesn't include a timestamp or includes an invalid timestamp.
>
> For example if you're using a GELF input and the GELF messages contain a 
> valid timestamp field, that timestamp is being used as message timestamp 
> in Graylog.
>
>
> Is there a workaround?
>>
>
> What exactly is the problem you're trying to solve? 
>
> Cheers,
> Jochen
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/34c8485f-08d8-4d94-905a-d126ced35e09%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] graylog2 timestamp not from application log message

[Wayne]

Hi

I am new to graylog2 and I am having an issue with the timestamp that is 
displayed in each message.

I understand that the timestamp reflects the time that graylog imported the 
log messages, and not the timestamp associated with the application log 
message. For example, if I send a log file from my application server to 
graylog server, the timestamp of my application log message is a different 
field (when extracted) in graylog UI

I was able to configure my application log message timestamp to be date 
type and search query have to be formulated to reflect the time zone 
difference since the "now" is going to be the UTC time. So I will have an 
awkward query like as follow (to query the latest 5 minute time frame)

"filter": {
"bool": {
  "must": {
"range": {
  "mytimestampDate": {
"from": "now-4h-5m",
"to": "now-4h",  
"include_lower": true,
"include_upper": true
  }
}
  }
}
  }

and NOT

"filter": {
"bool": {
  "must": {
"range": {
  "mytimestampDate": {
"from": "now-5m",
"to": "now",  
"include_lower": true,
"include_upper": true
  }
}
  }
}
  }

The BEST solution is to replace/overwrite the timestamp of the graylog 
server with the timestamp of the application log message that is shipped 
over to graylog2. This is because the web interface is using the timestamp 
to do query. 

I was able to do it with Logstash by using a date filter, and I was able to 
do it with Fluentd by using a plugin. Both worked beautifully. However, I 
have not found a solution for graylog2.

Is there a workaround?

Thanks

Wayne

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/24c9a97e-f979-4b0d-833b-6a012b2ab78e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.