[graylog2] Re: A question about clustering

[Frederic Desjarlais]

I'm assuming you're referring to Graylog Server "clustering" -- and not 
MongoDB or ElasticSearch.

If "high availability" is important to you, then I'd suggest having at 
least 2 Graylog Server processes running across as many availability zones 
as possible (e.g. different racks in a data center, etc.)

You'll also want to ensure that whatever is feeding your Graylog Servers 
can re-establish a "connection" (a term I'll use loosely due to UDP being a 
possible protocol) in the event of a failure with the Graylog Server it 
initially connects with.

Lastly, you should ensure that you measure the performance (e.g. CPU, disk, 
memory, throughput, etc.) of each Graylog Server node to ensure you have 
enough capacity.  You could also take a look 
at https://www.graylog.org/tools/sizing-estimator for some capacity 
planning help (but this shouldn't replace collecting performance metrics 
into your monitoring system).

In terms of 'hard limits' to the number of devices which can connect to a 
Graylog server, it depends on which input protocol is used (e.g. TCP or 
UDP).  For TCP-based connections, the number of file descriptors allocated 
to the Graylog Server process will determine this limit.

HTH,
Frederic




On Tuesday, June 28, 2016 at 10:30:19 AM UTC-7, Jamie P wrote:
>
> I have looked this group over and did some Google searches to no avail. 
>  My question is, at what point do you consider using a clustered setup vs. 
> a single server instance?  I know it's based off of how many servers and 
> devices will be reporting to the server but I can't find any info that 
> suggests "Well if you have this many devices and servers reporting in then 
> you should consider having this many nodes, and this many graylog 
> instances, etc., etc."
>
> I want to make sure that I build out the correct solution.  I don't want 
> to go overkill and over estimate, but I don't want to under estimate as 
> well.  Any documentation or websites discussing this would be most helpful. 
>  Thanks.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/c69368e3-7f36-42b7-b927-a3a121e4b97b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Kafka output plugin for Graylog 2.x?

[Frederic Desjarlais]

Is anyone aware of a Kafka output plugin for Graylog 2.x?  If not, is 
Graylog itself considering creating/maintaining such a plugin in the near 
future?

We're considering building one ourselves, but we'd like to ensure one 
doesn't already exist (or in the works).  We didn't find such a plugin in 
the Marketplace (https://marketplace.graylog.org/addons?search=kafka), nor 
via a regular Google query.

Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/9390bcfb-40d1-4c8d-8b16-28e70307a568%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: graylog server warning every 5-30 minutes

[Frederic Desjarlais]

Would it make sense to increase the 'stale_master_timeout' setting to 
something like 5 minutes?  What would be the issues to consider with a 
large cluster (say 32 Graylog Server nodes) having this set at 5 minutes 
(instead of 2000ms)?

My understanding is that the master is only needed to run some 
'periodicals' (such as ElasticSearch index rotations) -- and having 'no 
master' for several minutes isn't a problem.  Is this understanding correct?

Thanks,
Frederic



On Tuesday, June 21, 2016 at 2:59:22 AM UTC-7, Jochen Schalanda wrote:
>
> Hi Ariel,
>
> just for reference, I'll paraphrase the explanation from IRC:
>
> Each Graylog node "registers" itself (node id, URI to the Graylog REST 
>> API, timestamp of the last heartbeat) in MongoDB (see the nodes 
>> collection). The timeout/cleanup interval is quite aggressive (2s, see 
>> stale_master_timeout 
>> ),
>>  
>> so if your system clock is off by a minute or so, the information in 
>> MongoDB will be considered stale and the node is trying to re-register 
>> itself.
>
>
>
> Cheers,
> Jochen
>
> On Monday, 20 June 2016 18:13:52 UTC+2, Ariel Godinez wrote:
>>
>> Hello,
>>
>> I am getting the following related warnings from the graylog server every 
>> 5 to 30 minutes. 
>>
>> Warning (from graylog system messages page) : *Notification condition 
>> [NO_MASTER] has been fixed.*
>> Warning (from graylog server logs): *WARN : 
>> org.graylog2.periodical.NodePingThread - Did not find meta info of this 
>> node. Re-registering.*
>>
>> Upon googling these warnings I saw that multiple people were able to get 
>> these warnings to stop after installing NTP and synchronizing their 
>> system(s).
>>
>> I am running a single node configuration ( *my graylog server.conf: 
>> is_master = true* ) ,have installed NTP, and configured it. Graylog is 
>> working as expected but I just wanted to see if anyone had an idea as to 
>> what might be causing these annoying warnings and how I could get them to 
>> stop. 
>>
>> Any input would be much appreciated.
>>
>> System:
>> Oracle Linux Server release 6.5
>> Red Hat Enterprise Linux Server release 6.5 (Santiago)
>>
>> Thanks,
>> Ariel Godinez
>>
>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/1383fc1e-90ac-4aa1-a698-be95ea62ff08%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Need help on Graylog 3 node cluster

[Frederic Desjarlais]

Hi,

This appears to be more of an ElasticSearch question, rather than a 
Graylog-specific question.

I'm going to guess that your ElasticSearch cluster doesn't have any 
(primary) shards replicated to the other nodes in the cluster.  When you 
brought down one node, these shards were "lost" and ElasticSearch tells you 
this by setting the cluster status to 'red'.

Graylog's default setting for ElasticSearch replicas (e.g. from 
https://github.com/Graylog2/graylog2-server/blob/2.0/misc/graylog.conf) is:
---
elasticsearch_replicas = 0
---

Try setting this to: "elasticsearch_replicas = 1".

HTH,
Frederic




On Saturday, May 21, 2016 at 9:16:09 AM UTC-7, HASIF M wrote:
>
> Hi All,
>
>
> Please help me to create a 3 node Graylog cluster. All Instances are 
> running on CentOs 6.5.
>
> Node 1:-
>
> Graylog 2.0.1
> Elasticsearch 2.3.1
> MongoDB replica Set
>
>
> Elastic config file:-
>
> cluster.name: graylog2
>
>  discovery.zen.ping.multicast.enabled: false
>  discovery.zen.ping.unicast.hosts: ["graylog1.local:9300", 
> "graylog2.local:9300", "graylog3.local:9300"]
>  discovery.zen.minimum_master_nodes: 2
>  network.host: 192.168.1.128
>
> Graylog config:-
>
> rest_listen_uri = http://0.0.0.0:12900/
> web_listen_uri = http://0.0.0.0:9000/
> elasticsearch_cluster_name = graylog2
> elasticsearch_discovery_zen_ping_unicast_hosts = graylog1.local:9300, 
> graylog2.local:9300, graylog3.local:9300
> elasticsearch_discovery_zen_ping_multicast_enabled = false
> elasticsearch_network_host = 192.168.1.128
> mongodb_uri = 
> mongodb://graylog1.local:27017,graylog2.local:27017,graylog3.local:27017/graylog2
>
>
> Node 2:-
>
> Graylog 2.0.1
> Elasticsearch 2.3.1
> MongoDB replica Set
>
> Elastic config:-
> cluster.name: graylog2
>
>  discovery.zen.ping.multicast.enabled: false
>  discovery.zen.ping.unicast.hosts: ["graylog1.local:9300", 
> "graylog2.local:9300", "graylog3.local:9300"]
>  discovery.zen.minimum_master_nodes: 2
>  network.host: 192.168.1.129
>
> Graylog config:-
>
> rest_listen_uri = http://0.0.0.0:12900/
> web_listen_uri = http://0.0.0.0:9000/
> elasticsearch_cluster_name = graylog2
> elasticsearch_discovery_zen_ping_unicast_hosts = graylog1.local:9300, 
> graylog2.local:9300, graylog3.local:9300
> elasticsearch_discovery_zen_ping_multicast_enabled = false
> elasticsearch_network_host = 192.168.1.129
> mongodb_uri = 
> mongodb://graylog1.local:27017,graylog2.local:27017,graylog3.local:27017/graylog2
>
>
> Node 3:-
>
> Graylog 2.0.1
> Elasticsearch 2.3.1
> MongoDB replica Set
>
>
> Elasitc config:-
> cluster.name: graylog2
>
>  discovery.zen.ping.multicast.enabled: false
>  discovery.zen.ping.unicast.hosts: ["graylog1.local:9300", 
> "graylog2.local:9300", "graylog3.local:9300"]
>  discovery.zen.minimum_master_nodes: 2
>  network.host: 192.168.1.130
>
> Graylog config:-
>
> rest_listen_uri = http://0.0.0.0:12900/
> web_listen_uri = http://0.0.0.0:9000/
> elasticsearch_cluster_name = graylog2
> elasticsearch_discovery_zen_ping_unicast_hosts = graylog1.local:9300, 
> graylog2.local:9300, graylog3.local:9300
> elasticsearch_discovery_zen_ping_multicast_enabled = false
> elasticsearch_network_host = 192.168.1.130
> mongodb_uri = 
> mongodb://graylog1.local:27017,graylog2.local:27017,graylog3.local:27017/graylog2
>
>
> ES Status when all three nodes are UP.
>
> [root@graylog1 ~]# curl -X GET '
> http://192.168.1.130:9200/_cluster/health?pretty=true'
> {
>   "cluster_name" : "graylog2",
>   "status" : "green",
>   "timed_out" : false,
>   "number_of_nodes" : 6,
>   "number_of_data_nodes" : 3,
>   "active_primary_shards" : 8,
>   "active_shards" : 8,
>   "relocating_shards" : 0,
>   "initializing_shards" : 0,
>   "unassigned_shards" : 0,
>   "delayed_unassigned_shards" : 0,
>   "number_of_pending_tasks" : 0,
>   "number_of_in_flight_fetch" : 0,
>   "task_max_waiting_in_queue_millis" : 0,
>   "active_shards_percent_as_number" : 100.0
> }
>
>
>
>
> i am trying to create HA cluster, but my problem is, if any one node goes 
> down my elasticsearch status showing RED and throwing below error in log 
> file.
>
> Status when any node goes down:-
>
> [root@graylog1 ~]# curl -X GET '
> http://192.168.1.130:9200/_cluster/health?pretty=true'
> {
>   "cluster_name" : "graylog2",
>   "status" : "red",
>   "timed_out" : false,
>   "number_of_nodes" : 5,
>   "number_of_data_nodes" : 2,
>   "active_primary_shards" : 5,
>   "active_shards" : 5,
>   "relocating_shards" : 0,
>   "initializing_shards" : 0,
>   "unassigned_shards" : 3,
>   "delayed_unassigned_shards" : 0,
>   "number_of_pending_tasks" : 0,
>   "number_of_in_flight_fetch" : 0,
>   "task_max_waiting_in_queue_millis" : 0,
>   "active_shards_percent_as_number" : 62.5
> }
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 

[graylog2] Re: Beats plugin from Graylog vs. sivasamyk

[Frederic Desjarlais]

Hi Jochen -- we've been using the 3rd party one made by Sivasamy Kaliappan 
with Graylog 2.0 (since alpha) without any issues.  I understand from 
Lennart that Graylog 2.0 was meant to provide backwards compatibility with 
1.x plugins -- so that's likely why it works.

Does the 'official' one (from Graylog) make use of any specific 2.x 
features/APIs (now or forthcoming)?  Things have been stable with the 3rd 
party plugin -- so we're hesitant to move.  That said, with the upcoming 
5.0 Beats, it's likely that changes may be needed at some point (e.g. input 
format changes, or similar).

Thanks,
Frederic



On Tuesday, May 17, 2016 at 3:52:13 AM UTC-7, Jochen Schalanda wrote:
>
> Hi Frederic,
>
> both plugins add support for the Elastic Beats platform to Graylog. The 
> 3rd party one made by Sivasamy Kaliappan supports Graylog 1.x, while the 
> official one provided by Graylog, Inc. supports Graylog 2.x and later.
>
> Cheers,
> Jochen
>
> On Friday, 13 May 2016 19:11:10 UTC+2, Frederic Desjarlais wrote:
>>
>> Hi,
>>
>> With Graylog 2.0.0 (pre-GA), we've been using the Beats plugin from 
>> https://github.com/sivasamyk/graylog-beats-plugin 
>> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fsivasamyk%2Fgraylog-beats-plugin=D=1=AFQjCNFmlKL5mNOI54z-ca0JaMz0KjgRIA>
>>  
>> and we recently noticed that Graylog now offers a Beats plugin at 
>> https://github.com/Graylog2/graylog-plugin-beats .
>>
>> Could someone describe the difference between these plugins?
>>
>> Thanks,
>> Frederic
>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/34b56853-9d09-4c38-89ac-637d3ee35a6f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Beats plugin from Graylog vs. sivasamyk

[Frederic Desjarlais]

Hi,

With Graylog 2.0.0 (pre-GA), we've been using the Beats plugin 
from https://github.com/sivasamyk/graylog-beats-plugin and we recently 
noticed that Graylog now offers a Beats plugin 
at https://github.com/Graylog2/graylog-plugin-beats .

Could someone describe the difference between these plugins?

Thanks,
Frederic


-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/3d764214-4eeb-423d-b1ce-b88eea6c3c35%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Re: how alpha series install on centos 6 ?

[Frederic Desjarlais]

We've been using Graylog 2.0.0 alpha-5 and now beta-1 -- hopefully my 
response below is helpful to you.

To be clear, by "alpha" I'm assuming you mean the new Graylog 2.0 alpha 
releases (e.g. alpha1, alpha2, ... , alpha5).  There is now (as of today) a 
"beta1" release of Graylog 2.0.


1) elasticsearch 1.7 version is fine ?
- No, ES 2.2.x


2) mongo db 2.4 version is fine ?
- Should be fine.  We're using 3.2.x, but my understanding is that older 
versions (e.g. 2.4) of mongodb are fine since only very basic functionality 
is needed.


3) graylog-server 1.3 version is fine ?
- SInce you want to use the "alpha" series, you'd want to go to 
graylog-server 2.0.0-alpha[1-5] or the newer (as of today) beta1.


4) graylog-web-gui 1.3 version is fine ?
- With Graylog 2.0, there is no more 'graylog-web-gui' process.


5) should i untar the alpha & start the service only?
- Yes, untar and try using bin/graylogctl (if running on Linux).


6) should graylog-server & console service should remain start ?
- I'm not sure I understand this question.  Only the graylog-server process 
is needed with 2.0.


7) shoud 3 services remain start together ( alpha, graylog-server & 
graylog-web-gu) ?
- As mentioned above, you only need 1 process (i.e. the graylog-server). 
 You can't have pre-2.0 processs in the mix.


I hope this helps.

- Frederic



On Thursday, March 24, 2016 at 9:47:39 AM UTC-7, Amit Sharma wrote:
>
> Hi team,
>
> please help me to sort out this issue.
>
> Thanks 
> Amit 
>
> On Mon, Feb 29, 2016 at 9:33 PM, Amit Sharma  > wrote:
>
>> Thanks for the information jochen
>>
>> I have few questions for upgrading to alpha..
>>
>> 1) elasticsearch 1.7 version is fine ?
>> 2) mongo db 2.4 version is fine ?
>> 3) graylog-server 1.3 version is fine ?
>> 4) graylog-web-gui 1.3 version is fine ?Hi
>> 5) should i untar the alpha & start the service only?
>> 6) should graylog-server & console service should remain start ?
>> 7) shoud 3 services remain start together ( alpha, graylog-server & 
>> graylog-web-gu) ?
>>
>> Thanks 
>> Amit
>>
>>
>>
>>
>>
>> On 29 Feb 2016 7:26 pm, "Jochen Schalanda" > > wrote:
>>
>>> Hi Amit,
>>>
>>> we currently don't provide OS packages (DEB or RPM) of the Graylog 2.0.0 
>>> alpha versions or comprehensive upgrade instructions, so you're pretty much 
>>> on your own.
>>>
>>> Feel free to fetch the tar-ball from 
>>> https://www.graylog.org/pages/download_twodotoh and install it manually 
>>> on your system as described in 
>>> http://docs.graylog.org/en/2.0/pages/installation/manual_setup.html.
>>>
>>>
>>> Cheers,
>>> Jochen
>>>
>>> On Monday, 29 February 2016 13:38:00 UTC+1, amit@kaarya.com wrote:

 Hi Guys

 what is need to be done for installing alpha on centos 6 ?

 currently i am using graylog 1.0.3 & graylog-web-interface-1.3.0 
 version with elasticsearch 1.7 & Mongo 2.6 

 please suggest.

 Thanks 
 Amit 

>>> -- 
>>> You received this message because you are subscribed to a topic in the 
>>> Google Groups "Graylog Users" group.
>>> To unsubscribe from this topic, visit 
>>> https://groups.google.com/d/topic/graylog2/07McsYigFVA/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to 
>>> graylog2+u...@googlegroups.com .
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/graylog2/46a85227-d351-4fbb-90b7-7ef1ccfbdf6c%40googlegroups.com
>>>  
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/3b4f6c79-fe0a-466a-92f5-5411c6da5f18%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.