Re: [graylog2] Sidecar: When would you need more than one tag in a configuration?

[123Dev]

Agreed, the tag is confusing to us too. 
On Graylog, if I have 3 configurations.

   - Config1 - tag1 
   - Config2 - tag2
   - Config3 - tag3

On the collector side, I was wrongly expecting that if I set tag1 and tag2, 
the client would get both configurations.
But that didn't work

Because each configuration to be configured needs its own output, and the 
generated nxlog.conf did not get the two outputs or the auto-generated two 
routes.

I think Inputs, Outputs, Snippets and even routes (which cannot be defined 
now) need to be decoupled from a configuration and be allowed to be defined 
independently each in its own collections (similar to how the rules are 
done in pipelines) for convenience (you don't want to repeat for each 
config) and functionality (see below the nxlog internal example)
Then a configuration can pull one or many of the above components from the 
collections, and then applied to a collector.

If we don't decouple, then if we want to have nxlog internal logs logged to 
Graylog, we would need to define something like this in Snippets for each 
config (repeated and modified to match the output id)

Module  im_internal



  Path internal => 977ad164136aa0330cf2b422



and if the client has multiple tags, would it get multiple copies? single 
copy? which output? which route?

Sorry if we're understanding this thing totally wrong.

Thanks












-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/0eb49603-18b4-4e1e-98f7-02ccfd35ef08%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] debugging pipelines is... difficult

[Edmundo Alvarez]

I added this Github issue so you can track the issue I mentioned in point 
number 2: 
https://github.com/Graylog2/graylog-plugin-pipeline-processor/issues/46

Cheers,
Edmundo

> On 18 Jul 2016, at 10:51, Edmundo Alvarez  wrote:
> 
> I spent some time debugging the issue, and I found two of them:
> 
> 1. The when expression should be wrapped in a "to_bool" function, otherwise 
> the parser gets confused about it and replaces it with "false":
>
> to_bool(regex("[^0-9a-zA-Z]([0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+)[^0-9a-zA-Z].*[^0-9a-zA-Z]([0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+)[^0-9a-zA-Z]",to_string($message.message)).matches)
> 
> 2. There seems to be some problems when handling strings containing 
> backslashes. You need to escape them so they get parsed, but then the escape 
> character is still being used in the regular expression. I will investigate 
> further and keep you posted on that.
> 
> Cheers,
> Edmundo
> 
>> On 13 Jul 2016, at 12:31, Jason Haar  wrote:
>> 
>> 
>> On Mon, Jul 11, 2016 at 11:28 AM, Jason Haar  wrote:
>> If I take the regex I wrote in this rule (as per first email), replace '\\' 
>> with '\', then the regex works fine via egrep. It's a simple "when, do this" 
>> type statement: I can't see what's gone wrong in it
>> 
>> Oh - and thanks to your comment about the regex needing to match the entire 
>> line, I put ".*" at the beginning and end - but it made no difference. Still 
>> no Cisco syslog messages (as above) match
>> 
>> 
>> -- 
>> Cheers
>> 
>> Jason Haar
>> Information Security Manager, Trimble Navigation Ltd.
>> Phone: +1 408 481 8171
>> PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Graylog Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to graylog2+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/graylog2/CAFChrgJZng%2Bzc-iZ%2Bv73%2Bd8Q6YatVATaDtj2R%3Dd7sR9iXZfbHQ%40mail.gmail.com.
>> For more options, visit https://groups.google.com/d/optout.
> 

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/BA27A691-42D6-46BD-80B5-988211F400B3%40graylog.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Graylog Sidecar reports "unable to map property tags"

[Jeremy Farr]

Done.  https://github.com/Graylog2/collector-sidecar/issues/39

On Monday, July 18, 2016 at 3:35:36 AM UTC-5, Marius Sturm wrote:
>
> Hi,
> could you please create an issue for that over here: 
> https://github.com/Graylog2/collector-sidecar/issues
> Please add your collector_sidecar.yml file to the ticket.
>
> Thanks,
> Marius
>
>
> On 15 July 2016 at 20:25, Jeremy Farr  
> wrote:
>
>> So I'm using nxlog and I've installed the graylog sidecar.  I'm manually 
>> starting it with my configuration file so I can monitor it.  Just after 
>> reporting that nxlog is starting it gives a 400 error related to the 
>> property tags.  I've attached the screen shot. I've changed the tag and 
>> ensured it's the same as what I've got in the config on the graylog side. I 
>> am using the alpha release of the collector just FYI.
>>
>>
>> 
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Graylog Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to graylog2+u...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/graylog2/440c674f-b5ea-4315-9733-2e5c4429c41e%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> -- 
> Developer
>
> Tel.: +49 (0)40 609 452 077
> Fax.: +49 (0)40 609 452 078
>
> TORCH GmbH - A Graylog Company
> Poolstraße 21
> 20335 Hamburg
> Germany
>
> https://www.graylog.com 
>
> Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175
> Geschäftsführer: Lennart Koopmann (CEO)
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/924e3020-f765-4a97-b7f6-8f9841e64ef1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] debugging pipelines is... difficult

[Edmundo Alvarez]

I spent some time debugging the issue, and I found two of them:

1. The when expression should be wrapped in a "to_bool" function, otherwise the 
parser gets confused about it and replaces it with "false":

to_bool(regex("[^0-9a-zA-Z]([0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+)[^0-9a-zA-Z].*[^0-9a-zA-Z]([0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+)[^0-9a-zA-Z]",to_string($message.message)).matches)

2. There seems to be some problems when handling strings containing 
backslashes. You need to escape them so they get parsed, but then the escape 
character is still being used in the regular expression. I will investigate 
further and keep you posted on that.

Cheers,
Edmundo

> On 13 Jul 2016, at 12:31, Jason Haar  wrote:
> 
> 
> On Mon, Jul 11, 2016 at 11:28 AM, Jason Haar  wrote:
> If I take the regex I wrote in this rule (as per first email), replace '\\' 
> with '\', then the regex works fine via egrep. It's a simple "when, do this" 
> type statement: I can't see what's gone wrong in it
> 
> Oh - and thanks to your comment about the regex needing to match the entire 
> line, I put ".*" at the beginning and end - but it made no difference. Still 
> no Cisco syslog messages (as above) match
> 
> 
> -- 
> Cheers
> 
> Jason Haar
> Information Security Manager, Trimble Navigation Ltd.
> Phone: +1 408 481 8171
> PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Graylog Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to graylog2+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/graylog2/CAFChrgJZng%2Bzc-iZ%2Bv73%2Bd8Q6YatVATaDtj2R%3Dd7sR9iXZfbHQ%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/4A90E6BA-9C9C-4D9C-ADE8-787ADEFB1D54%40graylog.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Graylog Sidecar reports "unable to map property tags"

[Marius Sturm]

Hi,
could you please create an issue for that over here:
https://github.com/Graylog2/collector-sidecar/issues
Please add your collector_sidecar.yml file to the ticket.

Thanks,
Marius


On 15 July 2016 at 20:25, Jeremy Farr  wrote:

> So I'm using nxlog and I've installed the graylog sidecar.  I'm manually
> starting it with my configuration file so I can monitor it.  Just after
> reporting that nxlog is starting it gives a 400 error related to the
> property tags.  I've attached the screen shot. I've changed the tag and
> ensured it's the same as what I've got in the config on the graylog side. I
> am using the alpha release of the collector just FYI.
>
>
> 
>
> --
> You received this message because you are subscribed to the Google Groups
> "Graylog Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to graylog2+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/graylog2/440c674f-b5ea-4315-9733-2e5c4429c41e%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Developer

Tel.: +49 (0)40 609 452 077
Fax.: +49 (0)40 609 452 078

TORCH GmbH - A Graylog Company
Poolstraße 21
20335 Hamburg
Germany

https://www.graylog.com 

Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175
Geschäftsführer: Lennart Koopmann (CEO)

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/CAMqbBbLVFbjNUc%2BiN2fFveFaJ1s1zw3yzdsS8LproFfG-joHAQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Sidecar: When would you need more than one tag in a configuration?

[Marius Sturm]

Hi,
this depends on your tagging schema, let's say you have three classes of
machines 'database', 'application' and 'web_server' and you want to apply a
general configuration to all of them. E.g. a file input for
/var/log/messages, then you can create a configuration for that and use all
three tags.
But yes, to not get confused too much you should use at least on one side
just a single tag either the server side or the collector side.

Cheers,
Marius


On 16 July 2016 at 05:20, Werner van der Merwe 
wrote:

> Hi, this confuses me a bit.
>
> I understand a host can have multiple tags to combine multiple
> configurations, for example an apache server can have tags linux and apache.
>
> As I understand it, for this, two configurations will be created, one with
> a tag called linux and one with a tag called apache.
>
> So form that point I can understand it, but in what circumstances will a
> configuration have more than one tag?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Graylog Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to graylog2+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/graylog2/086e7094-c096-4ed7-8baf-dd96f1e1730c%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Developer

Tel.: +49 (0)40 609 452 077
Fax.: +49 (0)40 609 452 078

TORCH GmbH - A Graylog Company
Poolstraße 21
20335 Hamburg
Germany

https://www.graylog.com 

Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175
Geschäftsführer: Lennart Koopmann (CEO)

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/CAMqbBbKuptJMH1J0%3DkcAG%3DCAwn0X-4YHx6U33%3DH1o4dTWnrkJg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Unble to get graylog webinterface

[Jochen Schalanda]

Hi Anant,

please check that you're really hitting the web interface and not the 
Graylog REST API. The response you've mentioned 
({"type":"ApiError","message":"HTTP 
404 Not Found"}) comes from the Graylog REST and not from the Graylog web 
interface.

Also make sure that there is no reverse proxy running in front of Graylog 
or any transparent proxy on your network which may interfere with your 
requests.

Cheers,
Jochen

On Monday, 18 July 2016 09:32:32 UTC+2, Anant Sawant wrote:
>
> Hi Jochen,
>
> I am unable to get the web interface in the browser when I hit  "
> 127.0.0.1:9000" all I get is the following message on the screen 
> "{"type":"ApiError","message":HTTP 404 Not Found}", I am not sure what is 
> the cause of this error. The logs says "2016-07-18 12:36:19,915 INFO : 
> org.graylog2.initializers.WebInterfaceService - Started Web Interface at <
> http://127.0.0.1:9000/>" and also "2016-07-18 12:36:22,465 INFO : 
> org.graylog2.bootstrap.ServerBootstrap - Graylog server up and running.' So 
> the problem is I am not getting the UI of the "Graylog" only the above 
> mentioned message. I am attaching the Logs file of Graylog may be you can 
> find something which I am certainly missing to locate.
>
> Thanking in Advance
>
> Anant.
>
>
>
>
> On Friday, 15 July 2016 21:58:38 UTC+5:30, Jochen Schalanda wrote:
>>
>> Hi Anant,
>>
>> please describe first in detail what's not working for you.
>>
>> Cheers,
>> Jochen
>>
>> On Friday, 15 July 2016 15:20:24 UTC+2, Anant Sawant wrote:
>>>
>>> HI ,
>>>
>>> Thanks for such a quick replay!!
>>>
>>> I am already running the web interface on http://127.0.0.1:9000 
>>> 
>>>  
>>> and I have not upgraded Graylog to this version I have installed a fresh 
>>> release in completely new environment(machine). What do you mean by "if you 
>>> tried to get the root resource of the Graylog REST API (this will change in 
>>> Graylog 2.1.0)." sory but I do not understand.
>>> I went through the configuration again and compared it with your 
>>> suggested configurations "
>>> http://docs.graylog.org/en/2.0/pages/upgrade.html 
>>> 
>>>  
>>> and 
>>> http://docs.graylog.org/en/2.0/pages/configuration/web_interface.html; 
>>> but did not found any different. Is there anything we need to change in the 
>>> conf files. How do we overcome this?? I am attaching the conf files.
>>>
>>> Again Thanks in advance!!
>>>
>>> Anant
>>>
>>> On Friday, 15 July 2016 18:31:47 UTC+5:30, Jochen Schalanda wrote:

 Hi Anant,

 according to your logs, the Graylog REST API and the Graylog web 
 interface have been successfully started:

 2016-07-15 16:38:00,442 INFO : 
> org.graylog2.initializers.WebInterfaceService - Started Web Interface at <
> http://127.0.0.1:9000/>
> 2016-07-15 16:38:00,443 INFO : 
> org.graylog2.shared.initializers.RestApiService - Started REST API at <
> http://127.0.0.1:12900/>


 The response you've mentioned ({"type":"ApiError","message":"HTTP 404 
 Not Found"}) is totally normal if you tried to get the root resource 
 of the Graylog REST API (this will change in Graylog 2.1.0).

 Make sure to open http://127.0.0.1:9000/ 
 
  for 
 accessing the Graylog web interface and not http://127.0.0.1:12900/, 
 which is the Graylog REST API.

 Also make sure to read 
 http://docs.graylog.org/en/2.0/pages/upgrade.html 
 
  
 and 
 http://docs.graylog.org/en/2.0/pages/configuration/web_interface.html 
 when upgrading to Graylog 2.0.x.

 Cheers,
 Jochen


 On Friday, 15 July 2016 14:09:46 UTC+2, Anant Sawant wrote:
>
> Hi,
>
> I have installed graylog 2.0 ga on ubuntu 14.0.4 manually today. 
> Followed the following url for installation "
> http://docs.graylog.org/en/2.0/pages/installation/manual_setup.html; .
> After starting the graylogctl script for the first time and hitting 
> 127.0.0.1:9000 we are getting {"type":"ApiError","message":HTTP 404 
> Not Found} in the browser.
>
> The server logs says the "Graylog server up and running". I am unable 
> to locate the issue as I am not getting any error at the logs.Please give 
> me some advise to overcome this issue.
>
> I have attached the logs file for your perusal.
> Please  find the attachment.
>
> Thanks in Advance!!
>
> Anant.
>


-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To 

[graylog2] Re: Unble to get graylog webinterface

[Anant Sawant]

Hi Jochen,

I am unable to get the web interface in the browser when I hit  
"127.0.0.1:9000" all I get is the following message on the screen 
"{"type":"ApiError","message":HTTP 404 Not Found}", I am not sure what is 
the cause of this error. The logs says "2016-07-18 12:36:19,915 INFO : 
org.graylog2.initializers.WebInterfaceService - Started Web Interface at 
" and also "2016-07-18 12:36:22,465 INFO : 
org.graylog2.bootstrap.ServerBootstrap - Graylog server up and running.' So 
the problem is I am not getting the UI of the "Graylog" only the above 
mentioned message. I am attaching the Logs file of Graylog may be you can 
find something which I am certainly missing to locate.

Thanking in Advance

Anant.




On Friday, 15 July 2016 21:58:38 UTC+5:30, Jochen Schalanda wrote:
>
> Hi Anant,
>
> please describe first in detail what's not working for you.
>
> Cheers,
> Jochen
>
> On Friday, 15 July 2016 15:20:24 UTC+2, Anant Sawant wrote:
>>
>> HI ,
>>
>> Thanks for such a quick replay!!
>>
>> I am already running the web interface on http://127.0.0.1:9000 
>> 
>>  
>> and I have not upgraded Graylog to this version I have installed a fresh 
>> release in completely new environment(machine). What do you mean by "if you 
>> tried to get the root resource of the Graylog REST API (this will change in 
>> Graylog 2.1.0)." sory but I do not understand.
>> I went through the configuration again and compared it with your 
>> suggested configurations "
>> http://docs.graylog.org/en/2.0/pages/upgrade.html 
>> 
>>  
>> and http://docs.graylog.org/en/2.0/pages/configuration/web_interface.html; 
>> but did not found any different. Is there anything we need to change in the 
>> conf files. How do we overcome this?? I am attaching the conf files.
>>
>> Again Thanks in advance!!
>>
>> Anant
>>
>> On Friday, 15 July 2016 18:31:47 UTC+5:30, Jochen Schalanda wrote:
>>>
>>> Hi Anant,
>>>
>>> according to your logs, the Graylog REST API and the Graylog web 
>>> interface have been successfully started:
>>>
>>> 2016-07-15 16:38:00,442 INFO : 
 org.graylog2.initializers.WebInterfaceService - Started Web Interface at <
 http://127.0.0.1:9000/>
 2016-07-15 16:38:00,443 INFO : 
 org.graylog2.shared.initializers.RestApiService - Started REST API at <
 http://127.0.0.1:12900/>
>>>
>>>
>>> The response you've mentioned ({"type":"ApiError","message":"HTTP 404 
>>> Not Found"}) is totally normal if you tried to get the root resource of 
>>> the Graylog REST API (this will change in Graylog 2.1.0).
>>>
>>> Make sure to open http://127.0.0.1:9000/ 
>>> 
>>>  for 
>>> accessing the Graylog web interface and not http://127.0.0.1:12900/, 
>>> which is the Graylog REST API.
>>>
>>> Also make sure to read http://docs.graylog.org/en/2.0/pages/upgrade.html 
>>> 
>>>  
>>> and 
>>> http://docs.graylog.org/en/2.0/pages/configuration/web_interface.html 
>>> when upgrading to Graylog 2.0.x.
>>>
>>> Cheers,
>>> Jochen
>>>
>>>
>>> On Friday, 15 July 2016 14:09:46 UTC+2, Anant Sawant wrote:

 Hi,

 I have installed graylog 2.0 ga on ubuntu 14.0.4 manually today. 
 Followed the following url for installation "
 http://docs.graylog.org/en/2.0/pages/installation/manual_setup.html; .
 After starting the graylogctl script for the first time and hitting 
 127.0.0.1:9000 we are getting {"type":"ApiError","message":HTTP 404 
 Not Found} in the browser.

 The server logs says the "Graylog server up and running". I am unable 
 to locate the issue as I am not getting any error at the logs.Please give 
 me some advise to overcome this issue.

 I have attached the logs file for your perusal.
 Please  find the attachment.

 Thanks in Advance!!

 Anant.

>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/e9261dcf-85f7-47f3-ae26-ed34b7719754%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
ubuntu@ubuntu:/opt/graylog$ java -jar graylog.jar server
2016-07-18 12:36:13,480 main ERROR Appenders contains an invalid element or 
attribute "Memory"
2016-07-18 12:36:13,497 main ERROR Unable to locate appender 
"graylog-internal-logs" for logger config "root"
2016-07-18 12:36:13,622 INFO :