[graylog2] Re: Disk Journal / Kafka Input / Throttling

[Eli Jordan]

Thanks for the clarification Jochen.

Do you know if its possible to throttle the kafka input, so that messages 
are buffered in kafka rather than in GrayLogs internal journal? Enabling 
throttling on the input didn't seem to slow down the rate at which messages 
are consumed. (note: we are running 2.0.3)

On Tuesday, 19 July 2016 22:04:59 UTC+10, Jochen Schalanda wrote:
>
> Hi Eli,
>
> On Tuesday, 19 July 2016 13:18:49 UTC+2, Eli Jordan wrote:
>>
>> My understanding is that the disk journal is just an internal Kafka 
>> topic. Since we are already using Kafka to buffer messages, this seems 
>> redundant. (Also, since we are running graylog in docker the journal is 
>> transient without configuring appropriate docker volumes).
>>
>
> That's not quite correct. Graylog is using the journal implementation from 
> Apache Kafka internally but it's not a full-fledged Kafka broker, e. g. the 
> whole Kafka networking stack is missing (it's there for the Kafka client in 
> the Kafka inputs, of course).
>
> Cheers,
> Jochen
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/15bd2efe-2d3a-48e4-930f-e3055e07a02d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] mongod process using over 100% CPU slowing down graylog

[Ariel Godinez]

Hello,

I am running the single node setup below:

Graylog 2.0.3
MongoDB 3.2.7
Elasticsearch 2.3.3 
Red Hat Enterprise Linux Server 6.5
Java 8 
NXlog and Graylog Collector Sidecar for reading from local logs 

On average graylog is reading about 50 logs per second. MongoDB is not 
being used for any other services other than graylog. Yet, occasionally I 
notice that the system is hanging and proceed to do a  *$top *where I see 
that the mongod process is consuming well over 100% CPU. I'm wondering if 
the load is just to heavy or if there is something wrong with my setup that 
is causing mongod to overload. 

I am not seeing any warnings or errors in the graylog server logs or in the 
mongod.log file when I look after a slowdown has occurred. Any advice on 
how to further investigate would be much appreciated. 

Thanks,
Ari

   

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/da11761d-0e65-45d3-b16d-80c7d73263f0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Sidecar: When would you need more than one tag in a configuration?

[123Dev]

Thanks Marius for the explanation, and totally understand that the solution 
needs to be scalable and needs to address equally the small and big 
deployments.

Before Graylog Sidecar, all our client machines that were running nxlog, 
each and every one of them had their own configuration locally stored and 
managed.
Graylog Collector Sidecar is a great addition as it allows us to manage and 
update configurations centrally from Graylog.

If we look at it from the client machine (collector) perspective, it needs 
to advertise the services it has for potential logging.
Some Windows Examples:

   - Eventlog
   - Internal
   - MSSQL
   - IIS
   - IIS_Advanced
   - ...
   
The client would not advertise anything that needs to be kept private or is 
not available. (in Tags I suppose)
And that would be it from the client side.

On Graylog Server side.
Ideally we would want a flexible modular configuration that allows reuse of 
definitions without the need to copy repetitively from one config to 
another.
That is why I suggested decoupling of the Inputs, Outputs, Snippets, and 
Routes from Configurations and allowing to build various configurations by 
including elements from the 4 collections.

The idea is that the server knows what services are available on each 
client, and is at liberty (depending on business needs) to collect or stop 
collecting certain logs, route them to one output or to another and so 
forth all centrally from Graylog.
At no point the server should be able to collect logs for which the client 
hasn't advertised (authorized by tags)

The way it is now, if we want to gather logs of more services or less 
services from a specific client, we would need to modify the tags on the 
client (collector_sidecar.yml)
which kind of defeats the purpose of centralized control.

The way we got around this is really crude, we just added one tag per 
collector (host name)
created a configuration on the server with a matching tag, and dumped the 
entire nxlog.conf into snippet, making sure we modified the output to use 
the id defined in the output.

I'm certain we're not making the best use of graylog sidecar collector, as 
we have no modular control and all we have is managing the nxlog config 
remotely.

The above is not to suggest that Graylog should head in that direction, but 
rather provide a one customer use case so that you and your team 
understands how people are using the product, and design the best product 
based on common patterns / traits in usage.

One last note, if I want to force the sidecar to restart the nxlog service 
for whatever reasons (example in case it stopped sending data due to some 
broken network pipe) we have to edit the config and make dummy changes to 
force a reload.

it would be convenient if there was a way from the UI to tell the collector 
to reload the service.

Many thanks for not only providing such a wonderful product, but also 
actively responding to forum posts, supporting it and always looking for 
ways of improving it.
You folks are amazing.
Thanks




-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/34474028-966a-4a7c-9aaf-0fbe5cc441c3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Where does Chef keep the Web-Interface URI list?

[caleb]

Worked perfectly. 

Thanks, Marius!

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/fc863ca8-4a5e-445a-9b82-b36ab1da4b76%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Graylog compilation guideline

[Jochen Schalanda]

Hi Anant,

you basically just need Java 8 (we recommend using the latest Oracle JDK) 
and Maven 3 on your system. Everything else (e. g. Node.js) will be 
downloaded automatically.

Please refer to the .travis.yml 
 file 
which is being used on Travis CI to build Graylog from source.

Cheers,
Jochen

On Tuesday, 19 July 2016 15:51:03 UTC+2, Anant Sawant wrote:
>
>
> Hi,
>
> I am trying to compile Graylog 2.0.0 source code, I have previously 
> compiled Graylog 1.1.6 successfully. For compiling 1.1.6 server component I 
> used eclipse and for the web component I used "TypeSafe Activator", but I 
> am not sure about how to compile the 2.0.0 version as both components are 
> merged into single.
> I want to know what tools are required and  how to setup the compilation 
> environment for compiling 2.0.0. I have done an R and D on the same but 
> have not found any documentation for the compilation.  If any one knows 
> please share the method here, it would be very helpful. 
>
> Thanks in advance!!
>
> Anant.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/b322a05a-05bf-4d87-a454-3ae4d2c0e822%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Graylog compilation guideline

[Anant Sawant]

Hi,

I am trying to compile Graylog 2.0.0 source code, I have previously 
compiled Graylog 1.1.6 successfully. For compiling 1.1.6 server component I 
used eclipse and for the web component I used "TypeSafe Activator", but I 
am not sure about how to compile the 2.0.0 version as both components are 
merged into single.
I want to know what tools are required and  how to setup the compilation 
environment for compiling 2.0.0. I have done an R and D on the same but 
have not found any documentation for the compilation.  If any one knows 
please share the method here, it would be very helpful. 

Thanks in advance!!

Anant.

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/a6a7bdbc-b608-4cd2-bf2e-a528740d890d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Disk Journal / Kafka Input / Throttling

[Jochen Schalanda]

Hi Eli,

On Tuesday, 19 July 2016 13:18:49 UTC+2, Eli Jordan wrote:
>
> My understanding is that the disk journal is just an internal Kafka topic. 
> Since we are already using Kafka to buffer messages, this seems redundant. 
> (Also, since we are running graylog in docker the journal is transient 
> without configuring appropriate docker volumes).
>

That's not quite correct. Graylog is using the journal implementation from 
Apache Kafka internally but it's not a full-fledged Kafka broker, e. g. the 
whole Kafka networking stack is missing (it's there for the Kafka client in 
the Kafka inputs, of course).

Cheers,
Jochen

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/5561f4cc-2290-484f-b32e-c63a7d572957%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Disk Journal / Kafka Input / Throttling

[Eli Jordan]

Hi

We are using the Kafka input to pull logs into graylog, and when running load 
tests against our system, the graylog journal begins to fill up, since the 
elastic search indexing can't keep up. 

My understanding is that the disk journal is just an internal Kafka topic. 
Since we are already using Kafka to buffer messages, this seems redundant. 
(Also, since we are running graylog in docker the journal is transient without 
configuring appropriate docker volumes).

 We have plenty of storage in the Kafka cluster, so there should be no problem 
buffering messages until graylog can catch up. I was hoping that enabling 
throttling on the Kafka input would do this. I.e. Only read messages in as fast 
as they can be processed by elastic search. However, that doesn't seem to be 
the case. Even with throttling enabled, the disk journal still begins to fill 
up.

How should I go about achieving this behavior? Would disabling the disk journal 
help? Why isn't throttling slowing down the rate at which messages are being 
read in?

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/b61c1f16-441d-4e89-8568-a5002b884792%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Sidecar: When would you need more than one tag in a configuration?

[Werner vd Merwe]

Great point Marius!

Your explanation makes sense as well, I am beginning to better understand the 
intent. Great product and awesome idea, this final touches is the exciting last 
steps leading up to version 1.0!

I see this similarly to puppet and roles manifests. Having a tag per computer 
role and having the role made up of profile tags rather than, as you mentioned, 
the confusion of a mega bucket with a large amount of small puzzle pieces. 

Looking forward to spend some mind time on this when I am back in the country 
next month!


> On 19/07/2016, at 11:45, Marius Sturm  wrote:
> 
> Hi,
> thanks for the feedback, that's very helpful for us. Otherwise we never know 
> if the concepts are understood by the users or not.
> The current implementation is merging all configurations that are fetched 
> throug the provided tags. So let's say that your sidecar starts with three 
> tags, internally it's
> fetching all three configurations and merge them together to a single new 
> one. So you end up with a configuration that includes all 
> outputs/inputs/snippets from all three configurations. That said, the only 
> limitation is in the web interface. Currently you can't use an output from 
> one configuration with an input from another configuration.
> But that could be changed relatively easily.
> 
> Our goal is to find a solution that is intuitive and scales at the same time. 
> When you think of not having 1-5 configurations but maybe a few dozens. 
> Putting all inputs/output/snippets into one big bucket would be pretty 
> confusing imo. Losing the overview means you never know what will actually be 
> configured on the target system.
> 
> So I am very open to better ideas but they need to solve the problem for 
> bigger and smaller setups. Maybe there is some holy grail we missed till now.
> 
> Cheers,
> Marius
> 
> 
> 
>> On 18 July 2016 at 19:02, 123Dev  wrote:
>> Agreed, the tag is confusing to us too. 
>> On Graylog, if I have 3 configurations.
>> Config1 - tag1
>> Config2 - tag2
>> Config3 - tag3
>> On the collector side, I was wrongly expecting that if I set tag1 and tag2, 
>> the client would get both configurations.
>> But that didn't work
>> 
>> Because each configuration to be configured needs its own output, and the 
>> generated nxlog.conf did not get the two outputs or the auto-generated two 
>> routes.
>> 
>> I think Inputs, Outputs, Snippets and even routes (which cannot be defined 
>> now) need to be decoupled from a configuration and be allowed to be defined 
>> independently each in its own collections (similar to how the rules are done 
>> in pipelines) for convenience (you don't want to repeat for each config) and 
>> functionality (see below the nxlog internal example)
>> Then a configuration can pull one or many of the above components from the 
>> collections, and then applied to a collector.
>> 
>> If we don't decouple, then if we want to have nxlog internal logs logged to 
>> Graylog, we would need to define something like this in Snippets for each 
>> config (repeated and modified to match the output id)
>> 
>> Module  im_internal
>> 
>> 
>> 
>>   Path internal => 977ad164136aa0330cf2b422
>> 
>> 
>> 
>> and if the client has multiple tags, would it get multiple copies? single 
>> copy? which output? which route?
>> 
>> Sorry if we're understanding this thing totally wrong.
>> 
>> Thanks
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Graylog Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to graylog2+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/graylog2/0eb49603-18b4-4e1e-98f7-02ccfd35ef08%40googlegroups.com.
>> 
>> For more options, visit https://groups.google.com/d/optout.
> 
> 
> 
> -- 
> Developer
> 
> Tel.: +49 (0)40 609 452 077
> Fax.: +49 (0)40 609 452 078
> 
> TORCH GmbH - A Graylog Company
> Poolstraße 21
> 20335 Hamburg
> Germany
> 
> https://www.graylog.com
> 
> Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175
> Geschäftsführer: Lennart Koopmann (CEO)
> -- 
> You received this message because you are subscribed to a topic in the Google 
> Groups "Graylog Users" group.
> To unsubscribe from this topic, visit 
> https://groups.google.com/d/topic/graylog2/KL6kn_4hm6k/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to 
> graylog2+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/graylog2/CAMqbBbJK8TLr6JuwQUx-gj8UjDuwFrZ84O5UqRtAxb%2B5t--0NA%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 

[graylog2] Re: Configure Graylog WebInterface on a dedicated server

[Jochen Schalanda]

Hi,

you can disable the web interface on one of the two servers with the 
web_enable configuration 
setting: 
https://github.com/Graylog2/graylog2-server/blob/2.0.3/misc/graylog.conf#L80-L82

Cheers,
Jochen

On Tuesday, 19 July 2016 11:27:04 UTC+2, sangh wrote:
>
> So if i have 2 server. each one will have its own web interface ?? it is 
> not pratical when searching for logs
>
>
>
>
> Le mardi 19 juillet 2016 11:15:24 UTC+2, Jochen Schalanda a écrit :
>>
>> Hi,
>>
>> no, Graylog 2.x currently doesn't allow running only the web interface.
>>
>> Cheers,
>> Jochen
>>
>> On Tuesday, 19 July 2016 11:10:47 UTC+2, sangh wrote:
>>>
>>> Hi,
>>>
>>> I am using two graylog server with a load balancer. i want to install 
>>> the web interface along with the load balancer. With Graylog 2.0 is no 
>>> longer possible to run only the web inetrface. Is there a way to do it ??
>>>
>>> Thanks 
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/d362c116-ef5b-4871-a2cc-01ea07beaf8e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Sidecar: When would you need more than one tag in a configuration?

[Marius Sturm]

Hi,
thanks for the feedback, that's very helpful for us. Otherwise we never
know if the concepts are understood by the users or not.
The current implementation is merging all configurations that are fetched
throug the provided tags. So let's say that your sidecar starts with three
tags, internally it's
fetching all three configurations and merge them together to a single new
one. So you end up with a configuration that includes all
outputs/inputs/snippets from all three configurations. That said, the only
limitation is in the web interface. Currently you can't use an output from
one configuration with an input from another configuration.
But that could be changed relatively easily.

Our goal is to find a solution that is intuitive and scales at the same
time. When you think of not having 1-5 configurations but maybe a few
dozens. Putting all inputs/output/snippets into one big bucket would be
pretty confusing imo. Losing the overview means you never know what will
actually be configured on the target system.

So I am very open to better ideas but they need to solve the problem for
bigger and smaller setups. Maybe there is some holy grail we missed till
now.

Cheers,
Marius



On 18 July 2016 at 19:02, 123Dev  wrote:

> Agreed, the tag is confusing to us too.
> On Graylog, if I have 3 configurations.
>
>- Config1 - tag1
>- Config2 - tag2
>- Config3 - tag3
>
> On the collector side, I was wrongly expecting that if I set tag1 and
> tag2, the client would get both configurations.
> But that didn't work
>
> Because each configuration to be configured needs its own output, and the
> generated nxlog.conf did not get the two outputs or the auto-generated two
> routes.
>
> I think Inputs, Outputs, Snippets and even routes (which cannot be defined
> now) need to be decoupled from a configuration and be allowed to be defined
> independently each in its own collections (similar to how the rules are
> done in pipelines) for convenience (you don't want to repeat for each
> config) and functionality (see below the nxlog internal example)
> Then a configuration can pull one or many of the above components from the
> collections, and then applied to a collector.
>
> If we don't decouple, then if we want to have nxlog internal logs logged
> to Graylog, we would need to define something like this in Snippets for
> each config (repeated and modified to match the output id)
> 
> Module  im_internal
> 
>
> 
>   Path internal => 977ad164136aa0330cf2b422
> 
>
>
> and if the client has multiple tags, would it get multiple copies? single
> copy? which output? which route?
>
> Sorry if we're understanding this thing totally wrong.
>
> Thanks
>
>
>
>
>
>
>
>
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Graylog Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to graylog2+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/graylog2/0eb49603-18b4-4e1e-98f7-02ccfd35ef08%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Developer

Tel.: +49 (0)40 609 452 077
Fax.: +49 (0)40 609 452 078

TORCH GmbH - A Graylog Company
Poolstraße 21
20335 Hamburg
Germany

https://www.graylog.com 

Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175
Geschäftsführer: Lennart Koopmann (CEO)

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/CAMqbBbJK8TLr6JuwQUx-gj8UjDuwFrZ84O5UqRtAxb%2B5t--0NA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Configure Graylog WebInterface on a dedicated server

[sangh]

So if i have 2 server. each one will have its own web interface ?? it is 
not pratical when searching for logs




Le mardi 19 juillet 2016 11:15:24 UTC+2, Jochen Schalanda a écrit :
>
> Hi,
>
> no, Graylog 2.x currently doesn't allow running only the web interface.
>
> Cheers,
> Jochen
>
> On Tuesday, 19 July 2016 11:10:47 UTC+2, sangh wrote:
>>
>> Hi,
>>
>> I am using two graylog server with a load balancer. i want to install the 
>> web interface along with the load balancer. With Graylog 2.0 is no longer 
>> possible to run only the web inetrface. Is there a way to do it ??
>>
>> Thanks 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/851bc1dc-8887-4abe-86be-5a73b2c1dea1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Configure Graylog WebInterface on a dedicated server

[Jochen Schalanda]

Hi,

no, Graylog 2.x currently doesn't allow running only the web interface.

Cheers,
Jochen

On Tuesday, 19 July 2016 11:10:47 UTC+2, sangh wrote:
>
> Hi,
>
> I am using two graylog server with a load balancer. i want to install the 
> web interface along with the load balancer. With Graylog 2.0 is no longer 
> possible to run only the web inetrface. Is there a way to do it ??
>
> Thanks 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/9fd07860-5754-4764-8d06-3d104b8d441b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Configure Graylog WebInterface on a dedicated server

[sangh]

Hi,

I am using two graylog server with a load balancer. i want to install the 
web interface along with the load balancer. With Graylog 2.0 is no longer 
possible to run only the web inetrface. Is there a way to do it ??

Thanks 

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/4e1f1ac6-468f-4b7a-a519-1fd643f5493e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.