[graylog2] Re: can not search googlebot

[celtar]

Hi Frank,

thanks for the tip "field analyze", that was the right way. Elasticsearch 
uses per default dynamic Mapping as described here 
https://www.elastic.co/guide/en/elasticsearch/guide/2.x/dynamic-mapping.html 
.

I can use different type of fix mapping e.g in Elasticsearch, maybe i use 
Logstash or use the easy way in graylog (use an extractor as seen above).

We test this and get our solution to work.

Thanks and have nice day
John

PS: yes we activate leading wildcards and highlighting in graylog (testing 
performance is ok)

Am Dienstag, 14. Februar 2017 10:42:48 UTC+1 schrieb Frank Engler:
>
> Am Montag, 13. Februar 2017, 23:54:41 schrieb celtar: 
> > agent = (Original Message) :  "Mozilla/5.0 (compatible; Googlebot/2.1; 
> > +http://www.google.com/bot.html)" 
>
> > 1. Search = Input AND agent:*Googlebot* = result none found 
> > 2. Search = InputAND agent:*Googleb* = result none found 
> > 3. Search = Input AND agent:*Google*  = graylog result = ok but i 
> think 
> > graylog only found the string google from www.google.com 
> > 
> > I try different ways to search Google* or *Google  Google "Googlebot". 
> > Every time it is the same result. Google was found but not the String 
> > Googlbot 
> > 
> > Is there any Syntax failure in searching or maybe it is not possible. 
> > 
> > Do i have to use another Inputfilter  (like Logstash)? 
> > 
> > The same problem have we with different search string (not only 
> apache-gelf 
> > module). 
> > 
> > Thanks for helping and have a great day 
> > 
> > :) John Celtar 
>
> Did you allow leading wildcards for searches in graylog.conf? 
> Did you enable an analyzer for the agent field in the elasticsearch 
> template? 
>
>
> Frank 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/19187eee-10d7-4041-89e1-e3d849839212%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Incorrect Graylog Cluster details

[Paweł Karoluk]

Hi Jochen, You're right but there is another problem
I have tried to enable rest_transport_uri with "public IP" but it couldn't 
bind to Interface with port 9000 or 12900. I have SeLinux na IPtables 
disabled so it's not a problem, maybe something else

# netstat -tlpn | grep java
tcp0  0 :::127.0.0.1:9000  :::* 
LISTEN  62396/java  
tcp0  0 :::10.0.0.1:9200:::*   
  LISTEN  62396/java  
tcp0  0 :::10.0.0.1:9300:::*   
  LISTEN  62396/java 


My current config:

rest_listen_uri = http://127.0.0.1:9000/api/
rest_transport_uri = http://10.0.0.1:9000/api/

web_listen_uri = http://127.0.0.1:9000/
web_endpoint_uri = https://graylog1.local/api/

HAproxy config:
https://graylog1.local/ -> 127.0.0.1:9000

I will appreciate any help

Cheers!


Hi, I have two node Graylog Cluster and as you can see there is some wrong 
> with cluster config:
>
>
> *GET /api/system/cluster/nodes*
>
> {
> nodes: [
> {
> cluster_id: "6701202c-a9fe-42d2-8d5a-015acf66fbfa",
> node_id: "5f596ebf-a988-4c08-858e-67d38a3e483b",
> type: "server",
> transport_address: "http://127.0.0.1:9000/api/;,
> last_seen: "2017-02-10T00:45:30.000Z",
> short_node_id: "5f596ebf",
> hostname: "analog1.local",
> is_master: true
> },
> {
> cluster_id: "6701202c-a9fe-42d2-8d5a-015acf66fbfa",
> node_id: "8be9e293-f60b-40c6-a0e6-8af6d617eb1a",
> type: "server",
> transport_address: "http://127.0.0.1:9000/api/;,
> last_seen: "2017-02-10T00:45:30.000Z",
> short_node_id: "8be9e293",
> hostname: "analog2.local",
> is_master: false
> }
> ],
> total: 2
> }
>
>
> *GET /api/cluster*
>
> {
> 5f596ebf-a988-4c08-858e-67d38a3e483b: {
> facility: "graylog-server",
> codename: "Smuttynose",
> node_id: "5f596ebf-a988-4c08-858e-67d38a3e483b",
> cluster_id: "6701202c-a9fe-42d2-8d5a-015acf66fbfa",
> version: "2.1.3+040d371",
> started_at: "2017-02-10T00:27:13.101Z",
> hostname: "analog1.local",
> lifecycle: "running",
> lb_status: "alive",
> timezone: "Europe/Warsaw",
> operating_system: "Linux 2.6.32-642.13.1.el6.x86_64",
> is_processing: true
> },
> 8be9e293-f60b-40c6-a0e6-8af6d617eb1a: {
> facility: "graylog-server",
> codename: "Smuttynose",
> node_id: "5f596ebf-a988-4c08-858e-67d38a3e483b",
> cluster_id: "6701202c-a9fe-42d2-8d5a-015acf66fbfa",
> version: "2.1.3+040d371",
> started_at: "2017-02-10T00:27:13.101Z",
> hostname: "analog1.local",
> lifecycle: "running",
> lb_status: "alive",
> timezone: "Europe/Warsaw",
> operating_system: "Linux 2.6.32-642.13.1.el6.x86_64",
> is_processing: true
> }
> }
>
>
> In /api/cluster I supposed to get two different node_id and hostname, but 
> hostnames are the same. As the result when I want to check the 
> /system/nodes I got dubbed stats only of one host. The real HeapSize of 
> analog2 is only 2GB (img: analog2-system-nodes 
> ) not 4GB as analog1 - master node 
> (img: analog1-system-nodes ).
>
>
> MongoDB and ES Cluster are external and shared for both hosts.
>
>
> Thanks Guys
>
>
> Pawel
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/e73fcf7f-68bf-4c4b-a4f3-5649af8adfd7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] How to upgrade Graylog 2.1 > 2.2 ?

[dheffem]

I looked here  http://docs.graylog.org/en/2.2/pages/upgrade.html  and don't 
see any directions for upgrading Graylog 2.1  to 2.2. A Stackoverflow 
post[1] mentions backing up /etc/gralog2.conf and simply untarring the new 
graylog. Is this the correct upgrade path?  I've already posted this 
question by accident to the SELKS group so I have ruled out that I've 
likely missed something completely obvious. 

Thanks

 [1] 
http://stackoverflow.com/questions/25438095/how-can-i-upgrade-graylog2-to-a-newer-version

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/a8034597-15c2-4937-ab6c-052a7d89b6d3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: [ANN] Graylog 2.2.0 has been released

[Bill Murrin]

Graylog Team,

Congratulations on the release of 2.2.0. Can't wait to take it for a spin! 
:-)

-Bill

On Tuesday, February 14, 2017 at 5:07:13 AM UTC-10, Jochen Schalanda wrote:
>
> Hi everyone,
>
> I'm proud to announce the GA release of Graylog 2.2.0!
>
> We've put a lot of work into this release to bring you interesting 
> features like improved retention and rotation (index sets) and enhanced 
> alerting.
>
> You can find the release notes for Graylog 2.2.0 at:
>
> https://www.graylog.org/blog/88-announcing-graylog-v2-2-0
>
>
> If you have any questions about the new release of Graylog, don't hesitate 
> to get into one of our community support channels: 
> https://www.graylog.org/community-support
>
> And of course we're also offering professional support services for the 
> latest and greatest version of Graylog: 
> https://www.graylog.org/professional-support
>
>
> Previous release notes:
>
>- https://www.graylog.org/blog/77-announcing-graylog-2-2-0-beta-2
>- https://www.graylog.org/blog/78-announcing-graylog-v2-2-0-beta-3
>- https://www.graylog.org/blog/79-announcing-graylog-v2-2-0-beta-4
>- https://www.graylog.org/blog/80-announcing-graylog-v2-2-0-beta-5
>- https://www.graylog.org/blog/81-announcing-graylog-v2-2-0-beta-6
>- https://www.graylog.org/blog/85-announcing-graylog-v2-2-0-rc-1
>
>
> Cheers,
> Jochen (in the name of the Graylog team)
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/75aa3880-8fe1-4fc2-b9b6-2ee960bf3510%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] ElasticSearch Shards getting stuck in recovery

[karb]

Hello Graylog Users,

We're seeing a strange issue with our Graylog deployment. Things generally 
seem to work fine, except that on average once a day our ElasticSearch 
cluster will go yellow or red. We have our nodes distributed across two 
datacenters and the issue seems to happen following a brief network 
partition, at which point shards are dropped from an index. When their node 
re-joins the cluster the shards never reach 'active' state and we see 
errors like:

Caused by: java.lang.IllegalStateException: try to recover [graylog_208][2] 
from primary shard with sync id but number of docs differ...
We can manually recover by dropping all replicas and then bringing them 
back up or by manually rerouting a primary shard to a node, but if left be 
it will never recover on its own. Googling around seems to indicate this is 
a known issue in older versions of ElasticSearch with network partitioning 
(e.g. here  and here 
) but that it should 
be fixed (or at least improved) as of 5.0, but this doesn't help us when 
using Graylog and I can't find any mention of a backport.

My specific question here is: Have others run into this issue and overcome 
it using Graylog/ElasticSearch 2.3?
The ability to distribute across datacenters is important for us and I 
imagine much used by other people, so I have a hard time believing we're 
just that unlucky.

Any ideas/pointers/help would be much appreciated.

Thanks,
Kellen

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/71899341-9ff6-4711-949d-4fba86da7f19%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Remove field using extractos

[Rui Goncalves]

Thanks Jochen.

I'm looking at graylog pipelines docs, but I think I'm really confused :-/

I've created a pipeline with one rule that extracts key=value pairs:

rule "Extract K=V"
when true
then
set_fields(key_value(to_string($message.message)));
end

Then I've created a stream of messages, where only messages with the 
intended format will pass through. After that, I've connected the stream to 
the pipeline. Executing the "Simulate processing" function, fields get 
extracted as expected. However, graylog keeps indexing the original message 
into elasticsearch!

What am I missing? I think there must be something that I'm missing, 
because we can route the same message to multiple streams. If it worked the 
way I'm thinking, we would end up with duplicated messages on elasticsearch.

I've also looked at stream outputs, but there's no ES output. :-/

Can you shed some light here please?

Thanks.


On Tuesday, February 14, 2017 at 3:03:06 PM UTC, Jochen Schalanda wrote:
>
> Hi Rui,
>
> On Tuesday, 14 February 2017 13:15:13 UTC+1, Rui Goncalves wrote:
>>
>> Why it's not possible to remove a field from the received message using 
>> extractors?
>>
>
> This was a deliberate decision at the time to prevent people from 
> wondering why some field didn't exist anymore due to stacked or complicated 
> extractors.
>  
>
>> However it's in an experimental phase (with potential stability and 
>> performance issues) and it seems overkill for doing something so simple as 
>> dropping a field.
>>
>
> The message processing pipelines aren't experimental anymore in Graylog 
> 2.2.0.
>
> Cheers,
> Jochen
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/c4e6b5b6-be05-4461-a167-0418bbdafc5e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] [ANN] Graylog 2.2.0 has been released

[Jochen Schalanda]

Hi everyone,

I'm proud to announce the GA release of Graylog 2.2.0!

We've put a lot of work into this release to bring you interesting features 
like improved retention and rotation (index sets) and enhanced alerting.

You can find the release notes for Graylog 2.2.0 at:

https://www.graylog.org/blog/88-announcing-graylog-v2-2-0


If you have any questions about the new release of Graylog, don't hesitate 
to get into one of our community support 
channels: https://www.graylog.org/community-support

And of course we're also offering professional support services for the 
latest and greatest version of 
Graylog: https://www.graylog.org/professional-support


Previous release notes:

   - https://www.graylog.org/blog/77-announcing-graylog-2-2-0-beta-2
   - https://www.graylog.org/blog/78-announcing-graylog-v2-2-0-beta-3
   - https://www.graylog.org/blog/79-announcing-graylog-v2-2-0-beta-4
   - https://www.graylog.org/blog/80-announcing-graylog-v2-2-0-beta-5
   - https://www.graylog.org/blog/81-announcing-graylog-v2-2-0-beta-6
   - https://www.graylog.org/blog/85-announcing-graylog-v2-2-0-rc-1
   

Cheers,
Jochen (in the name of the Graylog team)

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/a64e6dfd-e83a-4f2e-84d9-477fe905ca58%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Remove field using extractos

[Jochen Schalanda]

Hi Rui,

On Tuesday, 14 February 2017 13:15:13 UTC+1, Rui Goncalves wrote:
>
> Why it's not possible to remove a field from the received message using 
> extractors?
>

This was a deliberate decision at the time to prevent people from wondering 
why some field didn't exist anymore due to stacked or complicated 
extractors.
 

> However it's in an experimental phase (with potential stability and 
> performance issues) and it seems overkill for doing something so simple as 
> dropping a field.
>

The message processing pipelines aren't experimental anymore in Graylog 
2.2.0.

Cheers,
Jochen

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/681e4875-a2db-48dc-a7c3-5fdebb263657%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Remove field using extractos

[Rui Goncalves]

Hi all.

I'm receiving messages following the pattern key=value. I'd like to set the 
value of two of the received keys on graylog standard fields, namely 
"message" and "timestamp" and discard the original fields completely.

I can use the "key=value" converter, then "copy" the original field's 
content and set the value on graylog fields, however the unnecessary fields 
(msg, and time) remain on the log. Using the cut option causes the field 
value to be set with the value "fullyCutByExtractor".

The question is: Why it's not possible to remove a field from the received 
message using extractors?  I believe having to drop fields is something 
usual.?! I've checked the docs (haven't tried yet), and it's possible to 
accomplish what I want using the pipelines feature. However it's in an 
experimental phase (with potential stability and performance issues) and it 
seems overkill for doing something so simple as dropping a field.

Thanks.

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/6bd8d93a-9dfe-448e-93bc-9644e589a221%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: can not search googlebot

[Frank Engler]

Am Montag, 13. Februar 2017, 23:54:41 schrieb celtar:
> agent = (Original Message) :  "Mozilla/5.0 (compatible; Googlebot/2.1; 
> +http://www.google.com/bot.html)"

> 1. Search = Input AND agent:*Googlebot* = result none found
> 2. Search = InputAND agent:*Googleb* = result none found
> 3. Search = Input AND agent:*Google*  = graylog result = ok but i think
> graylog only found the string google from www.google.com
> 
> I try different ways to search Google* or *Google  Google "Googlebot".
> Every time it is the same result. Google was found but not the String
> Googlbot
> 
> Is there any Syntax failure in searching or maybe it is not possible.
> 
> Do i have to use another Inputfilter  (like Logstash)?
> 
> The same problem have we with different search string (not only apache-gelf
> module).
> 
> Thanks for helping and have a great day
> 
> :) John Celtar

Did you allow leading wildcards for searches in graylog.conf?
Did you enable an analyzer for the agent field in the elasticsearch template?


Frank

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/2445367.505P3CerCN%40studio.engler.invalid.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: can not search googlebot

[celtar]

Hi,

i found it. I have to use Extractors.

http://docs.graylog.org/en/2.1/pages/extractors.html

Thx
john celtar

Am Dienstag, 14. Februar 2017 08:54:42 UTC+1 schrieb celtar:
>
> Hi,
>
> we use graylog 2.1.2 with the apache-gelf Module from the marketplace.
>
> If we try to search "Googlebot" in this string (type agent:)
>
> agent = (Original Message) :  "Mozilla/5.0 (compatible; Googlebot/2.1; +
> http://www.google.com/bot.html)"
>
> 1. Search = Input AND agent:*Googlebot* = result none found
>
> 2. Search = InputAND agent:*Googleb* = result none found
>
> 3. Search = Input AND agent:*Google*  = graylog result = ok but i 
> think graylog only found the string google from www.google.com
>
> I try different ways to search Google* or *Google  Google "Googlebot". 
> Every time it is the same result. Google was found but not the String 
> Googlbot
>
> Is there any Syntax failure in searching or maybe it is not possible.
>
> Do i have to use another Inputfilter  (like Logstash)?
>
> The same problem have we with different search string (not only 
> apache-gelf module).
>
> Thanks for helping and have a great day
>
> :) John Celtar
>
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/ac00d003-5071-4db7-b8e3-3cbb6c161603%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.