[graylog2] Replace Timestamp
Hello Guys,
Is posibble replace the Timestamp?. Graylog create a timestamp when
the logs are send to Graylog, but I need replace the timestamp by date
inside in logs.
I'm using Nxlog and here is my input:
Module im_file
File 'E:\\Hadoop\\ParsedLogs\\*.*'
SavePos TRUE
Exec if $raw_event =~/^((\d+-\d+-\d+) (\d+:\d+:\d+)),\d+
((?:INFO|ERROR|WARN)) (org.apache.hadoop.\w+.\w+): (.*)/ \
{ \
$date = $1; \
$time = $2; \
$Timestamp = parsedate($date + " " + $time); \
$CStatus = $3; \
$Process = $4; \
$Process_result = $5; \
to_json(); \
} \
else \
{ \
drop(); \
}
But now I have a field named Timestamp, and the Histogram show me the
Timestamp when the log was put in the Graylog. Maybe any way to replace the
Timestamp in the Histogram?.
I attached an exmple about I told you.
Thank you.
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/63d60185-739d-4b29-a93b-a62735a8a996%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[graylog2] Timestamp field depecrated.
Hello Guys,
I tried migrate my indexes from elasticsearch 1.7 to 2.0, but I
have a lot problem with _timestamp field, and I can't migrate the indexes
because I have this error:
"error" : {
"root_cause" : [ {
"type" : "snapshot_restore_exception",
"reason" : "[my_backup:snapshot_1] cannot restore index [graylog2_19]
because it cannot be upgraded"
} ],
"type" : "snapshot_restore_exception",
"reason" : "[my_backup:snapshot_1] cannot restore index [graylog2_19]
because it cannot be upgraded",
"caused_by" : {
"type" : "illegal_state_exception",
"reason" : "unable to upgrade the mappings for the index
[graylog2_19], reason: [Mapper for [_timestamp] conflicts with existing
mapping in other types:\n[mapper [_timestamp] is used by multiple types.
Set update_all_types to true to update [format] across all types.]]",
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "Mapper for [_timestamp] conflicts with existing mapping
in other types:\n[mapper [_timestamp] is used by multiple types. Set
update_all_types to true to update [format] across all types.]"
}
}
},
"status" : 500
This is index number 19 and I have 50...I can't lose my data, anyone knows
how to fix it?.
Link elasticsearch about this topic:
https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-timestamp-field.html
Thank you.
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/0dd7f687-f0ea-4180-a0a8-8af070d4ed0a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: Remove old source
Hello,
I'm opening this old treath because I have the same problem.
I used the same command to delete every message with source as target.
For example:
curl -XGET 'http://10.101.81.199:9200/graylog2_20/message/_search?pretty'
My output is :
{
"_index" : "graylog2_20",
"_type" : "message",
"_id" : "9d8cd406-605f-11e5-943e-005056a9199b",
"_score" : 1.0,
"_source":{"gl2_source_node":"297d10be-8e9e-4021-9ab6-deedd27202ce",
"s-ip":"10.101.250.209","time-taken":73,"csUser-Agent":
"Jakarta+Commons-HttpClient/3.1","EventReceivedTime":"2015-09-21 08:54:28",
"date":"2015-09-21","request_time":"12:54:24","version":"1.1","s-port":443,
"timestamp":"2015-09-21 12:54:28.000","SourceModuleName":"iis","time":
"12:54:24","level":6,"_id":"9d8cd406-605f-11e5-943e-005056a9199b",
"gl2_source_input":"5585b15184ae398b735b8d36","c-ip":"64.145.75.146",
"SourceModuleType":"im_file","full_message":"2015-09-21 12:54:24
10.101.250.209 GET /p1/clients/6035757/populationData
Jakarta+Commons-HttpClient/3.1 200 0 0 73","cs-uri-stem":
"/p1/clients/6035757/populationData","sc-win32-status":0,"cs-method":"GET",
"message":"2015-09-21 12:54:24 10.101.250.209 GET /p1/clients/6035757/popul"
,"sc-status":"200","SourceName":"IIS","sc-substatus":0,*"source":"SERVER-1"*
,"streams":[]}
So I want to delete every input with source: SERVER-1 in index graylog2_20.
I tried with the following command but the output is null, I'm testing with
XGET.
# curl -XGET 'http://10.101.81.199:9200/graylog2_20/messages/_query' -d '{
"query_string": {
"default_field" : "source",
"query": "SERVER-1"}}'
Output:
{"_index":"graylog2_20","_type":"messages","_id":"_query","found":false}
someone knows how to Delete by source?.
Thank you.
On Thursday, January 16, 2014 at 6:26:40 AM UTC-3, Jean-Luc Bassereau wrote:
>
> That looks something like this for me :
>
> curl -XDELETE 'http://127.0.0.1:9200/graylog2_*/message/_query' -d ' {
> "query_string" : { "default_field" : "host", "query" : "HOSTNAME" } }'
>
>
> 2014/1/16 Kay Röpke
>
>> Hi!
>>
>> You mean the ones listed on the "Sources" page?
>> Those are calculated from the messages in the current indices. Based on
>> your retention settings the hosts listed there will eventually go away.
>>
>> Graylog2 currently doesn't have a method to delete data, if you really
>> have to get rid of it, going to elasticsearch directly is your best bet at
>> this point:
>>
>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-delete-by-query.html
>>
>> Best,
>> Kay
>>
>> On Thursday, January 16, 2014 10:06:40 AM UTC+1, Martin Zeug wrote:
>>>
>>> Hi I installed rc1 - works great. But how to remove old sources not uses
>>> anymore?
>>>
>>> Greetings,
>>>
>>> Martin
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "graylog2" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to graylog2+u...@googlegroups.com .
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
>
>
> --
> Cordialement,
> Jean-Luc Bassereau
>
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/0e551aea-366a-48e4-af8d-5aacc1f39446%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[graylog2] Extractor for Puppet modified to free version
Hello,
I download the extractor Puppet from Marketplace, but this isn't
working to free version of Puppet (Puppet labs). I modified this extractor
and now is working as well.
Copy /paste my edited extractor for puppet:
"extractor_type": "regex",
"order": 0,
"source_field": "message",
"target_field": "puppetError",
"title": "Puppet error - DevOps"
},
{
"condition_type": "regex",
"condition_value":
"^\\d+-+\\d+-\\d+\\s+\\d+:+\\d+:\\d+\\s+-\\d+\\s+Puppet+\\s+\\(+notice+\\)+:\\s+Finished+\\s+catalog+\\s+run+\\s+in+\\s+(\\d[0-9]{0,9}\\.[0-9]{0,9})+\\s+seconds",
"converters": [
{
"config": {},
"type": "numeric"
}
],
"cursor_strategy": "copy",
"extractor_config": {
"regex_value": "Finished catalog run in
((\\d[0-9]{0,9}\\.[0-9]{0,9})) seconds"
},
"extractor_type": "regex",
"order": 0,
"source_field": "full_message",
"target_field": "puppetRunTime",
"title": "Puppet: Puppet run time - DevOps"
},
{
"condition_type": "regex",
"condition_value":
"^\\d+-+\\d+-\\d+\\s+\\d+:+\\d+:\\d+\\s+-\\d+\\s+Puppet+\\s+\\(+info+\\)+:\\s+Applying+\\s+configuration+\\s+version+\\s+'(\\d[0-9]{0,9})+'",
"converters": [
{
"config": {},
"type": "numeric"
}
],
"cursor_strategy": "copy",
"extractor_config": {
"regex_value":
"^\\d+-+\\d+-\\d+\\s+\\d+:+\\d+:\\d+\\s+-\\d+\\s+Puppet+\\s+\\(+info+\\)+:\\s+Applying+\\s+configuration+\\s+version+\\s+'(\\d[0-9]{0,9})+'"
},
"extractor_type": "regex",
"order": 0,
"source_field": "full_message",
"target_field": "puppetConfigVersion",
"title": "Puppet: Puppet Configuration Version - DevOps"
},
{
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/a57d937d-cc36-4747-aaf2-d55a9346c769%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Extractor for Puppet modified to free version
I forget the module to extract every name of Puppet modules:
"condition_type": "regex",
"condition_value":
"^\\d+-\\d+-\\d+\\s+\\d+:\\d+:\\d+\\s+-\\d+\\s+\\/+Stage+\\[+main+\\]+\\/(.*?)+\\/",
"converters": [],
"cursor_strategy": "copy",
"extractor_config": {
"index": 3,
"split_by": "/"
},
"extractor_type": "split_and_index",
"order": 0,
"source_field": "full_message",
"target_field": "puppetModule",
"title": "Puppet main Module name"
}
],
On Monday, September 7, 2015 at 2:23:57 PM UTC-3, Juan Andres Ramirez wrote:
>
> Hello,
>I download the extractor Puppet from Marketplace, but this isn't
> working to free version of Puppet (Puppet labs). I modified this extractor
> and now is working as well.
>Copy /paste my edited extractor for puppet:
>
> "extractor_type": "regex",
> "order": 0,
> "source_field": "message",
> "target_field": "puppetError",
> "title": "Puppet error - DevOps"
> },
> {
> "condition_type": "regex",
> "condition_value":
> "^\\d+-+\\d+-\\d+\\s+\\d+:+\\d+:\\d+\\s+-\\d+\\s+Puppet+\\s+\\(+notice+\\)+:\\s+Finished+\\s+catalog+\\s+run+\\s+in+\\s+(\\d[0-9]{0,9}\\.[0-9]{0,9})+\\s+seconds",
> "converters": [
> {
> "config": {},
> "type": "numeric"
> }
> ],
> "cursor_strategy": "copy",
> "extractor_config": {
> "regex_value": "Finished catalog run in
> ((\\d[0-9]{0,9}\\.[0-9]{0,9})) seconds"
> },
> "extractor_type": "regex",
> "order": 0,
> "source_field": "full_message",
> "target_field": "puppetRunTime",
> "title": "Puppet: Puppet run time - DevOps"
> },
> {
> "condition_type": "regex",
> "condition_value":
> "^\\d+-+\\d+-\\d+\\s+\\d+:+\\d+:\\d+\\s+-\\d+\\s+Puppet+\\s+\\(+info+\\)+:\\s+Applying+\\s+configuration+\\s+version+\\s+'(\\d[0-9]{0,9})+'",
> "converters": [
> {
> "config": {},
> "type": "numeric"
> }
> ],
> "cursor_strategy": "copy",
> "extractor_config": {
> "regex_value":
> "^\\d+-+\\d+-\\d+\\s+\\d+:+\\d+:\\d+\\s+-\\d+\\s+Puppet+\\s+\\(+info+\\)+:\\s+Applying+\\s+configuration+\\s+version+\\s+'(\\d[0-9]{0,9})+'"
> },
> "extractor_type": "regex",
> "order": 0,
> "source_field": "full_message",
> "target_field": "puppetConfigVersion",
> "title": "Puppet: Puppet Configuration Version - DevOps"
> },
> {
>
>
>
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/37fbf03d-ac85-47df-93b5-fa02f4b583ce%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Users and access on Global search
Understood, Thank you. On Tuesday, August 25, 2015 at 11:46:35 AM UTC-3, Kay Röpke wrote: Hi! It won’t be in 1.2, even though you could in theory grant the necessary permissions to roles. The main problem is that access global search requires more than just the search permissions in order to successfully use the search pages. Right now this is only possible via the REST API, have a look at the user API, specifically the permissions array. The additional permissions you want are: searches:keyword:*, searches:absolute:*, searches:relative:*”, These must be _in addition to_ the existing ones of the user, otherwise it will not work. In 1.2 you will be able to create a role with these permissions (via the API) and then assign them to users. Best, Kay On 25 Aug 2015, at 16:42, Juan Andres Ramirez jandr...@gmail.com javascript: wrote: Hello Guys, Read's users can't access on global search, this is a problem, because if you have a dashboard with a chart about any process, some time you need to see more details about this search. Anyone knows if in next versions the global search will be available for read's users?. Thank you. -- You received this message because you are subscribed to the Google Groups Graylog Users group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+u...@googlegroups.com javascript:. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/a5a3b69a-fa2d-4776-9937-8b32d8bbcf75%40googlegroups.com https://groups.google.com/d/msgid/graylog2/a5a3b69a-fa2d-4776-9937-8b32d8bbcf75%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Graylog Users group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/f1229cd0-a72a-4056-b50c-182ebfbd1f18%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Users and access on Global search
Hello Guys, Read's users can't access on global search, this is a problem, because if you have a dashboard with a chart about any process, some time you need to see more details about this search. Anyone knows if in next versions the global search will be available for read's users?. Thank you. -- You received this message because you are subscribed to the Google Groups Graylog Users group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/a5a3b69a-fa2d-4776-9937-8b32d8bbcf75%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: About shards
Understood, Thank you.
On Wednesday, July 22, 2015 at 7:13:14 PM UTC-3, Arie wrote:
Juan,
the part in de es config is done by the graylog ihmo, maybe even
overridden by it.
You can put a comment (#) in front of that.
And in elasticsearch conf I have this:
# index.number_of_shards: 1
# index.number_of_replicas: 0
Arie.
Op woensdag 22 juli 2015 15:03:45 UTC+2 schreef Juan Andres Ramirez:
Hello Arie,
In my graylog conf I have this:
elasticsearch_shards = 1
elasticsearch_replicas = 0
And in elasticsearch conf I have this:
index.number_of_shards: 1
index.number_of_replicas: 0
So why I have 16 shards in my cluster Health?, That is my question.
Thank you.
On Tuesday, July 21, 2015 at 6:25:54 PM UTC-3, Arie wrote:
Hi Juan,
IHMO for production having 4 ES nodes 4 shards can be fine. The data
will be shared on the 4 nodes leaving you
with 4 shards. (one on each node) Turning replicas to 1 wil create 1
replicated shard for each one there is. This gives you a backup
and improves search speed.
This is only in count for the new index that will be created if our
setup is already running, but there are some commands
in es that can make that happen for the current index.
see:
https://www.elastic.co/guide/en/elasticsearch/reference/1.6/indices-update-settings.html
Choosing the number of replicas:
https://www.elastic.co/guide/en/elasticsearch/guide/current/replica-shards.html
So for backup take one replica, for speed improvement choose 3 when
having 4 nodes.
Every node is than capable of serving a search request.
A.
Op dinsdag 21 juli 2015 15:19:21 UTC+2 schreef Juan Andres Ramirez:
The cluster health:
{
cluster_name : elasticsearch,
status : yellow,
timed_out : false,
number_of_nodes : 2,
number_of_data_nodes : 1,
active_primary_shards : 16,
active_shards : 16,
relocating_shards : 0,
initializing_shards : 0,
unassigned_shards : 1,
delayed_unassigned_shards : 0,
number_of_pending_tasks : 0,
number_of_in_flight_fetch : 0
}
On Tuesday, July 21, 2015 at 9:54:23 AM UTC-3, Juan Andres Ramirez
wrote:
Hello guys,
I was searching the answer in this group and in the web, but I
can't found the answer.
1- Graylog create 1 shard per indice?, so in this moment I have 17
shards and in my config I have :
elasticsearch_shards = 1
elasticsearch_replicas = 0
So I'm in development phase, I don't need replicas.
2- If I will change in config elasticsearch_shards = 2 , then I'm
going to have 34 shards 2 per index?.
My last question, If I'm going to create an Elasticsearch a cluster
with 4 nodes and change the setup elasticsearch_replicas = 1 , I'm going
to
have 17 shard in every node automatically?
I have problem to know how to work the elasticsearch cluster and the
configuration to failover.
Thank you.
--
You received this message because you are subscribed to the Google Groups
graylog2 group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: About shards
I got it.
Thank you !!
On Wednesday, July 22, 2015 at 12:55:20 PM UTC-3, Jochen Schalanda wrote:
Hi Juan,
so there are 15 indices managed by Graylog in your Elasticsearch cluster
each with 1 shard and no replica (+1 index used by Kibana with 1 shard and
1 replica). That's exactly what you've configured (according to the
settings you've shared before).
If you want to use less indices, take a look at the rotation_strategy and
related settings in your Graylog configuration (
https://github.com/Graylog2/graylog2-server/blob/1.1.4/misc/graylog2.conf#L89-127)
– I wouldn't recommend doing this, though.
Cheers,
Jochen
On Wednesday, 22 July 2015 16:59:31 UTC+2, Juan Andres Ramirez wrote:
Hi Jochen,
Here the output:
index shard prirep state docs store ipnode
graylog2_14 0 p STARTED 78979192 40.5gb 127.0.0.1 domain.com
graylog2_8 0 p STARTED 50004879 24.6gb 127.0.0.1 domain.com
graylog2_13 0 p STARTED12676 48.8gb 127.0.0.1 domain.com
graylog2_9 0 p STARTED 50005637 24.6gb 127.0.0.1 domain.com
graylog2_6 0 p STARTED 50001244 24.6gb 127.0.0.1 domain.com
graylog2_7 0 p STARTED 68925168 34.8gb 127.0.0.1 domain.com
graylog2_4 0 p STARTED 5795 24.9gb 127.0.0.1 domain.com
graylog2_5 0 p STARTED 50002618 24.6gb 127.0.0.1 domain.com
graylog2_2 0 p STARTED 2875 10.2gb 127.0.0.1 domain.com
graylog2_3 0 p STARTED 50003323 24.5gb 127.0.0.1 domain.com
graylog2_0 0 p STARTED 20001987 10.3gb 127.0.0.1 domain.com
graylog2_1 0 p STARTED 20003375 9.9gb 127.0.0.1 domain.com
.kibana 0 p STARTED 13 165.2kb 127.0.0.1 domain.com
.kibana 0 r UNASSIGNED
graylog2_10 0 p STARTED 70002642 34.6gb 127.0.0.1 domain.com
graylog2_11 0 p STARTED 90007163 44.1gb 127.0.0.1 domain.com
graylog2_12 0 p STARTED17815 48.3gb 127.0.0.1 domain.com
Thank you !!
On Wednesday, July 22, 2015 at 11:50:16 AM UTC-3, Jochen Schalanda wrote:
Hi Juan,
please post the output of the following command (replace 127.0.0.1 with
the IP address or hostname of one of your Elasticsearch nodes):
curl 'http://127.0.0.1:9200/_cat/shards?v'
Cheers,
Jochen
On Wednesday, 22 July 2015 15:03:45 UTC+2, Juan Andres Ramirez wrote:
Hello Arie,
In my graylog conf I have this:
elasticsearch_shards = 1
elasticsearch_replicas = 0
And in elasticsearch conf I have this:
index.number_of_shards: 1
index.number_of_replicas: 0
So why I have 16 shards in my cluster Health?, That is my question.
Thank you.
On Tuesday, July 21, 2015 at 6:25:54 PM UTC-3, Arie wrote:
Hi Juan,
IHMO for production having 4 ES nodes 4 shards can be fine. The data
will be shared on the 4 nodes leaving you
with 4 shards. (one on each node) Turning replicas to 1 wil create 1
replicated shard for each one there is. This gives you a backup
and improves search speed.
This is only in count for the new index that will be created if our
setup is already running, but there are some commands
in es that can make that happen for the current index.
see:
https://www.elastic.co/guide/en/elasticsearch/reference/1.6/indices-update-settings.html
Choosing the number of replicas:
https://www.elastic.co/guide/en/elasticsearch/guide/current/replica-shards.html
So for backup take one replica, for speed improvement choose 3 when
having 4 nodes.
Every node is than capable of serving a search request.
A.
Op dinsdag 21 juli 2015 15:19:21 UTC+2 schreef Juan Andres Ramirez:
The cluster health:
{
cluster_name : elasticsearch,
status : yellow,
timed_out : false,
number_of_nodes : 2,
number_of_data_nodes : 1,
active_primary_shards : 16,
active_shards : 16,
relocating_shards : 0,
initializing_shards : 0,
unassigned_shards : 1,
delayed_unassigned_shards : 0,
number_of_pending_tasks : 0,
number_of_in_flight_fetch : 0
}
On Tuesday, July 21, 2015 at 9:54:23 AM UTC-3, Juan Andres Ramirez
wrote:
Hello guys,
I was searching the answer in this group and in the web, but
I can't found the answer.
1- Graylog create 1 shard per indice?, so in this moment I have 17
shards and in my config I have :
elasticsearch_shards = 1
elasticsearch_replicas = 0
So I'm in development phase, I don't need replicas.
2- If I will change in config elasticsearch_shards = 2 , then I'm
going to have 34 shards 2 per index?.
My last question, If I'm going to create an Elasticsearch a cluster
with 4 nodes and change the setup elasticsearch_replicas = 1 , I'm
going to
have 17 shard in every node automatically?
I have problem to know how to work the elasticsearch cluster and the
configuration to failover.
Thank you
[graylog2] Re: About shards
Hi Jochen,
Here the output:
index shard prirep state docs store ipnode
graylog2_14 0 p STARTED 78979192 40.5gb 127.0.0.1 domain.com
graylog2_8 0 p STARTED 50004879 24.6gb 127.0.0.1 domain.com
graylog2_13 0 p STARTED12676 48.8gb 127.0.0.1 domain.com
graylog2_9 0 p STARTED 50005637 24.6gb 127.0.0.1 domain.com
graylog2_6 0 p STARTED 50001244 24.6gb 127.0.0.1 domain.com
graylog2_7 0 p STARTED 68925168 34.8gb 127.0.0.1 domain.com
graylog2_4 0 p STARTED 5795 24.9gb 127.0.0.1 domain.com
graylog2_5 0 p STARTED 50002618 24.6gb 127.0.0.1 domain.com
graylog2_2 0 p STARTED 2875 10.2gb 127.0.0.1 domain.com
graylog2_3 0 p STARTED 50003323 24.5gb 127.0.0.1 domain.com
graylog2_0 0 p STARTED 20001987 10.3gb 127.0.0.1 domain.com
graylog2_1 0 p STARTED 20003375 9.9gb 127.0.0.1 domain.com
.kibana 0 p STARTED 13 165.2kb 127.0.0.1 domain.com
.kibana 0 r UNASSIGNED
graylog2_10 0 p STARTED 70002642 34.6gb 127.0.0.1 domain.com
graylog2_11 0 p STARTED 90007163 44.1gb 127.0.0.1 domain.com
graylog2_12 0 p STARTED17815 48.3gb 127.0.0.1 domain.com
Thank you !!
On Wednesday, July 22, 2015 at 11:50:16 AM UTC-3, Jochen Schalanda wrote:
Hi Juan,
please post the output of the following command (replace 127.0.0.1 with
the IP address or hostname of one of your Elasticsearch nodes):
curl 'http://127.0.0.1:9200/_cat/shards?v'
Cheers,
Jochen
On Wednesday, 22 July 2015 15:03:45 UTC+2, Juan Andres Ramirez wrote:
Hello Arie,
In my graylog conf I have this:
elasticsearch_shards = 1
elasticsearch_replicas = 0
And in elasticsearch conf I have this:
index.number_of_shards: 1
index.number_of_replicas: 0
So why I have 16 shards in my cluster Health?, That is my question.
Thank you.
On Tuesday, July 21, 2015 at 6:25:54 PM UTC-3, Arie wrote:
Hi Juan,
IHMO for production having 4 ES nodes 4 shards can be fine. The data
will be shared on the 4 nodes leaving you
with 4 shards. (one on each node) Turning replicas to 1 wil create 1
replicated shard for each one there is. This gives you a backup
and improves search speed.
This is only in count for the new index that will be created if our
setup is already running, but there are some commands
in es that can make that happen for the current index.
see:
https://www.elastic.co/guide/en/elasticsearch/reference/1.6/indices-update-settings.html
Choosing the number of replicas:
https://www.elastic.co/guide/en/elasticsearch/guide/current/replica-shards.html
So for backup take one replica, for speed improvement choose 3 when
having 4 nodes.
Every node is than capable of serving a search request.
A.
Op dinsdag 21 juli 2015 15:19:21 UTC+2 schreef Juan Andres Ramirez:
The cluster health:
{
cluster_name : elasticsearch,
status : yellow,
timed_out : false,
number_of_nodes : 2,
number_of_data_nodes : 1,
active_primary_shards : 16,
active_shards : 16,
relocating_shards : 0,
initializing_shards : 0,
unassigned_shards : 1,
delayed_unassigned_shards : 0,
number_of_pending_tasks : 0,
number_of_in_flight_fetch : 0
}
On Tuesday, July 21, 2015 at 9:54:23 AM UTC-3, Juan Andres Ramirez
wrote:
Hello guys,
I was searching the answer in this group and in the web, but I
can't found the answer.
1- Graylog create 1 shard per indice?, so in this moment I have 17
shards and in my config I have :
elasticsearch_shards = 1
elasticsearch_replicas = 0
So I'm in development phase, I don't need replicas.
2- If I will change in config elasticsearch_shards = 2 , then I'm
going to have 34 shards 2 per index?.
My last question, If I'm going to create an Elasticsearch a cluster
with 4 nodes and change the setup elasticsearch_replicas = 1 , I'm going
to
have 17 shard in every node automatically?
I have problem to know how to work the elasticsearch cluster and the
configuration to failover.
Thank you.
--
You received this message because you are subscribed to the Google Groups
graylog2 group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[graylog2] About shards
Hello guys, I was searching the answer in this group and in the web, but I can't found the answer. 1- Graylog create 1 shard per indice?, so in this moment I have 17 shards and in my config I have : elasticsearch_shards = 1 elasticsearch_replicas = 0 So I'm in development phase, I don't need replicas. 2- If I will change in config elasticsearch_shards = 2 , then I'm going to have 34 shards 2 per index?. My last question, If I'm going to create an Elasticsearch a cluster with 4 nodes and change the setup elasticsearch_replicas = 1 , I'm going to have 17 shard in every node automatically? I have problem to know how to work the elasticsearch cluster and the configuration to failover. Thank you. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: About shards
The cluster health:
{
cluster_name : elasticsearch,
status : yellow,
timed_out : false,
number_of_nodes : 2,
number_of_data_nodes : 1,
active_primary_shards : 16,
active_shards : 16,
relocating_shards : 0,
initializing_shards : 0,
unassigned_shards : 1,
delayed_unassigned_shards : 0,
number_of_pending_tasks : 0,
number_of_in_flight_fetch : 0
}
On Tuesday, July 21, 2015 at 9:54:23 AM UTC-3, Juan Andres Ramirez wrote:
Hello guys,
I was searching the answer in this group and in the web, but I
can't found the answer.
1- Graylog create 1 shard per indice?, so in this moment I have 17 shards
and in my config I have :
elasticsearch_shards = 1
elasticsearch_replicas = 0
So I'm in development phase, I don't need replicas.
2- If I will change in config elasticsearch_shards = 2 , then I'm going to
have 34 shards 2 per index?.
My last question, If I'm going to create an Elasticsearch a cluster with 4
nodes and change the setup elasticsearch_replicas = 1 , I'm going to have
17 shard in every node automatically?
I have problem to know how to work the elasticsearch cluster and the
configuration to failover.
Thank you.
--
You received this message because you are subscribed to the Google Groups
graylog2 group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Alert when Graylog stop to receiving data from some server
I think I need bit help with this please. So I have 1 rule: source must match exactly serverName And I want the alarm triggered when this field doesn't has this value: serverName in last 5 min, but I'm lost with alerts of streams, I selected the next value, but nothing happend: Alert is triggered when the field source has a lower sum than 1 in the last 2 minutes. Grace period: 0 minutes. Including last message in alert notification Any other idea? Thank you. On Wednesday, July 15, 2015 at 10:39:49 AM UTC-3, Juan Andres Ramirez wrote: Hello guys, It is possible by any way?. I need create an alert when any server stop send data for the last 20 minutes, for example. Thank you. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Graylog and field source as ip address
FYI I fixed my problem was changing the input mode, so I replaced om_tcp by om_udp. Thank you. On Friday, July 10, 2015 at 5:46:39 AM UTC-3, Juan Andres Ramirez wrote: Hello guys, I was searching the solution for my problem but I can't found the answer. I have a server with graylog version: 1.1.3, connected to other server with Elasticsearch. I created a inputs type Raw/Plaintext TCP to get RabbitMQ logs from a Windows server 2008. The agent to get these logs is Nxlog. I recieved the logs as well, but the field source show me the ip address and not the hostname. I checked the server if got the dns from server, so I ran the following commands for checking it: [root@localhost ~]# host 10.101.250.119 119.250.101.10.in-addr.arpa domain name pointer cviaddzw12.office.xxx.com. 119.250.101.10.in-addr.arpa domain name pointer cviaddzw12.datacenter.xxx. com. [root@localhost ~]# dig -x 10.101.250.119 ; DiG 9.8.2rc1-RedHat-9.8.2-0.30.rc1.el6_6.3 -x 10.101.250.119 ;; global options: +cmd ;; Got answer: ;; -HEADER- opcode: QUERY, status: NOERROR, id: 10435 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;119.250.101.10.in-addr.arpa. IN PTR ;; ANSWER SECTION: 119.250.101.10.in-addr.arpa. 1200 INPTR cviaddzw12.datacenter.xxx. com. 119.250.101.10.in-addr.arpa. 1200 INPTR cviaddzw12.office.xxx.com. ;; Query time: 0 msec ;; SERVER: 10.101.1.52#53(10.101.1.52) ;; WHEN: Thu Jul 9 13:00:58 2015 ;; MSG SIZE rcvd: 125 I think the problem isn't the resolv DNS. Configuration Nxlog (extract) define SERVER serverName Extension fileop Module xm_fileop /Extension # Watch your own files. Input rabbitmq Module im_file File'C:\\rabbitmq\\log\\rabbit.log' SavePos TRUE Exec$Hostname = '%SERVER%'; Exec$Server = 'CVIADDZW12'; /Input Output out Module om_tcp Host10.101.81.190 Port /Output Route 1 Pathrabbitmq = out /Route I tried create an other field named $Server, but isn't works too. Anyone has any idea?. Thank you very much. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Graylog and field source as ip address
Hello guys, I was searching the solution for my problem but I can't found the answer. I have a server with graylog version: 1.1.3, connected to other server with Elasticsearch. I created a inputs type Raw/Plaintext TCP to get RabbitMQ logs from a Windows server 2008. The agent to get these logs is Nxlog. I recieved the logs as well, but the field source show me the ip address and not the hostname. I checked the server if got the dns from server, so I ran the following commands for checking it: [root@localhost ~]# host 10.101.250.119 119.250.101.10.in-addr.arpa domain name pointer cviaddzw12.office.xxx.com. 119.250.101.10.in-addr.arpa domain name pointer cviaddzw12.datacenter.xxx. com. [root@localhost ~]# dig -x 10.101.250.119 ; DiG 9.8.2rc1-RedHat-9.8.2-0.30.rc1.el6_6.3 -x 10.101.250.119 ;; global options: +cmd ;; Got answer: ;; -HEADER- opcode: QUERY, status: NOERROR, id: 10435 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;119.250.101.10.in-addr.arpa. IN PTR ;; ANSWER SECTION: 119.250.101.10.in-addr.arpa. 1200 INPTR cviaddzw12.datacenter.xxx. com. 119.250.101.10.in-addr.arpa. 1200 INPTR cviaddzw12.office.xxx.com. ;; Query time: 0 msec ;; SERVER: 10.101.1.52#53(10.101.1.52) ;; WHEN: Thu Jul 9 13:00:58 2015 ;; MSG SIZE rcvd: 125 I think the problem isn't the resolv DNS. Configuration Nxlog (extract) define SERVER serverName Extension fileop Module xm_fileop /Extension # Watch your own files. Input rabbitmq Module im_file File'C:\\rabbitmq\\log\\rabbit.log' SavePos TRUE Exec$Hostname = '%SERVER%'; Exec$Server = 'CVIADDZW12'; /Input Output out Module om_tcp Host10.101.81.190 Port /Output Route 1 Pathrabbitmq = out /Route I tried create an other field named $Server, but isn't works too. Anyone has any idea?. Thank you very much. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
